Effective separation of buildings from trees is a major challenge in image-based automatic building detection. This paper presents a three-step method for effective separation of buildings from trees using aerial imagery and lidar data. First, it uses cues such as height to remove objects of low height such as bushes, and width to exclude trees with small horizontal coverage. The height threshold is also used to generate a ground mask where buildings are found to be more separable than in so-called normalized DSM. Second, image entropy and color information are jointly applied to remove easily distinguishable trees. Finally, an innovative rule-based procedure is employed using the edge orientation histogram from the imagery to eliminate false positive candidates. The improved building detection algorithm has been tested on different test areas and it is shown that the algorithm offers high building detection rate in complex scenes which are hilly and densely vegetated.