Automatic 3D reconstruction of building roofs from remotely sensed data is important for many applications including automatic city modeling. This paper proposes a new method for automatic roof reconstruction using LIDAR (Light Detection And Ranging) data and multispectral imagery. Using the ground height from a DEM (Digital Elevation Model), the raw LIDAR points are separated into two groups. The first group contains the ground points that are exploited to constitute a 'ground mask'. The second group contains the non-ground points that are used to generate initial roof planes. The structural lines are extracted from the grey-scale version of the orthoimage and they are classified into several classes such as 'ground', 'ground with occasional tree', 'tree', 'roof edge' and 'roof ridge' using the ground mask, the NDVI image (from the multi-band orthoimage) and the entropy image (from the grey-scale orthoimage). The lines from the latter two classes are primarily used to fit initial planes to the neighbouring LIDAR points. Other image lines within the vicinity of an initial plane are selected to fit the boundary of the plane. Once the proper image lines are selected and others are discarded, the final plane is reconstructed using the selected lines. Experimental results show that the proposed method can handle irregular and large registration errors between the LIDAR data and orthoimagery.