Hyperspectral (HS) imaging and light detection and ranging (LiDAR) are widely used in remote sensing to acquire data from a same area of earth surface. HS image and LiDAR data contain complementary information of the target objects. Jointly using these two data modalities has great potential in land cover classification. In recent years, deep learning based fusion methods demonstrated promising performance on this task. However, how to better model the relationship of heterogeneous features from HS and LiDAR and their importance for the classification remains a challenging task. In this paper, we propose a spectral and spatial residual attention network for HS and LiDAR fusion and classification. A spectral residual attention module and a spatial residual attention module are designed in the network for better feature learning and fusion. Experiments on widely adopted Houston dataset demonstrate the superiority of the proposed method.