Automated building detection has been an active topic in photogrammetry and computer vision. One of the challenges is to effectively separate buildings from trees using aerial imagery and Lidar data. In cases where an adopted building detection technique cannot distinguish between these two classes of objects, the presence of trees in the scene can increase the rates of both false positives and false negatives in the building detection process. This paper presents an automatic building detection technique which exhibits improved separation of buildings from trees. In addition to using traditional features such as height, width and colour, the improved detector uses texture and edge orientation information from both Lidar and orthoimagery. Therefore, image entropy and colour information are jointly applied to remove easily distinguishable trees. Afterwards, a rule-based procedure using the edge orientation histogram from the imagery is followed to eliminate false positive candidates. The improved detector has been tested on a number of scenes from three different test areas. It is demonstrated that the algorithm performs well even in complex scenes and a 10% increase both in completeness and correctness has been achieved.