With the availability of high-resolution aerial imagery, we propose an attention-based dual-stream decoder architecture for weakly supervised solar panel mapping with the advantage of both reduced annotation costs and enhanced mapping performance. The proposed method unifies a target mapping branch (TMB) and a proximity detection branch (PDB). The TMB is developed based on cross-scale attention accumulation to discover desired objects and generate fine predictions, while the PDB is designed to estimate the proximity of the desired objects. Coupled with the dual-stream decoder, we also propose a subspace contrast loss to maximize the distance between the features of the desired objects and their proximity regions, so that precise object boundaries can be reserved. To reveal the effectiveness of the proposed method, we take the Australian Capital Territory, Australia as the study area and presented comprehensive experiments. Experimental results show that the proposed method outperforms the state-of-the-art approaches by a significant margin in mapping performance and presents a superior ability to identify object boundaries.