Convolutional neural networks (CNNs) with 3-D convolutional kernels are widely used for hyperspectral image (HSI) classification, which bring notable benefits in capturing joint spectral and spatial features. However, they suffer from poor computational efficiency, causing the low training/inference speed of the model. On the contrary, CNN-based methods with 1-D and 2-D kernels are efficient but mostly restricted to extracting either spectral or spatial features. Moreover, most CNN-based HSI classification frameworks are incapable of simultaneously taking advantage of residual and dense aggregations without over-allocating parameters or information loss for feature reusage. This article presents a novel attention-based lattice network (ALN) to overcome these shortcomings. The proposed 2-D lattice framework can effectively harness the advantages of residual and dense aggregations to achieve outstanding accuracy performance and computational efficiency simultaneously. Furthermore, the ALN employs a unique joint spectral-spatial attention mechanism to capture both spectral and spatial information effectively. In particular, a new pointwise spectral attention mechanism is adopted to fully capture spectral dependences for every pixel. Our extensive experimental investigation verifies the effectiveness and efficiency of the ALN architecture for HSI classification.