Pooling layers are an important part of convolutional neural networks (CNNs). They reduce the dimensionality of feature maps and pass salient information to subsequent layers. In this paper, we introduce a novel gradient-based feature pooling method that can down-sample feature maps while better preserving key information. This method considers the spatial gradient of the pixels within a pooling region as a key to select the most possible descriptive information in contrast to the current practice of existing methods that mostly rely on the pixel values. Extensive experiments on different benchmark image classification tasks and CNN architectures demonstrate that the proposed method achieves superior results over existing pooling approaches.