Rectilinear building outline generation from the point set of a building usually works in three steps. Boundary edges that constitute the building outline are first identified. A sequence of points is then traced from the edges to define the building boundary. Finally, lines are generated from the sequence of points and adjusted to form a regular building outline. Existing solutions have shortcomings in one or more of the following cases: identifying details along a concave shape, separate identification of a ‘hole’ inside the shape, proper boundary tracing, and preservation of detailed information along a regularized building outline. This article proposes new solutions to all three steps. By using the maximum point-to-point distance in the input data, the solution to the identification step properly detects the boundary edges for any type of shape and separately recognizes holes, if any, inside the shape. The proposed tracing algorithm divides boundary edges into segments, accurately obtains the sequence of points for each segment and then merges them, if necessary, to produce a single boundary for each shape. The regularization step proposes an improved corner and line extraction algorithm and adjusts the extracted lines with respect to the automatically determined principal directions of buildings. In order to evaluate the performance, an evaluation system that makes corner correspondences between an extracted building outline and its reference outline is also proposed. Experimental results show that the proposed solutions can preserve detail along the building boundary and offer high pixel-based completeness and geometric accuracy, even in low-density input data.