The convolutional neural network (CNN) is a feedforward neural network with deep structure and convolution operation. In the hyperspectral image (HSI) classification, CNN has demonstrated excellent performance in extracting spectral and spatial information. However, the inherent complexity and high dimension of HSIs still limit the performance of most neural network models. The powerful feature extraction ability of CNN is normally achieved by dozens or more layers, which brings a series of problems such as gradient vanishing, overfitting, and slow training speed. In order to address these problems, this article presents a CNN architecture-based stochastic depth residual network (SDRN), which is specially designed for HSI data. This model takes the original 3-D cube as the input and 3-D convolution is used to extract abundant spectral and spatial features through corresponding residual blocks. In order to reduce the training time, we adopt a stochastic depth strategy. For each small batch, a sublayer is randomly discarded by an identity function. During the testing stage, the residual network with complete depth is used. Experiments on three datasets and a comparison of the state-of-art methods show that SDRN has great advantages in accuracy and training time compared with state-of-the-art HSI classification methods.