This paper presents an automatic system for the detection of buildings from LIDAR data and multispectral imagery, which employs a threshold-free evaluation system that does not involve any thresholds based on human choice. Two binary masks are obtained from the LIDAR data: a ‘primary building mask’ and a ‘secondary building mask’. Line segments are extracted from around the primary building mask, the segments around trees being removed using the normalized difference vegetation index derived from orthorectified multispectral images. Initial building positions are obtained based on the remaining line segments. The complete buildings are detected from their initial positions using the two masks and multispectral images in the YIQ colour system. The proposed threshold-free evaluation system makes one-to-one correspondences using nearest centre distances between detected and reference buildings. A total of 15 indices are used to indicate object-based, pixel-based and geometric accuracy of the detected buildings. It is experimentally shown that the proposed technique can successfully detect rectilinear buildings, when assessed in terms of these indices including completeness, correctness and quality.