The rice leaf suffers from several bacterial, viral, or fungal diseases and these diseases reduce rice production significantly. To sustain rice demand for a vast population globally, the recognition of rice leaf diseases is crucially important. However, recognition of rice leaf disease is limited to the image backgrounds and image capture conditions. The convolutional neural network (CNN) based model is a hot research topic in the field of rice leaf disease recognition. But the existing CNN-based models drop in recognition rates severely on independent dataset and are limited to the learning of large scale network parameters. In this paper, we propose a novel CNN-based model to recognize rice leaf diseases by reducing the network parameters. Using a novel dataset of 4199 rice leaf disease images, a number of CNN-based models are trained to identify five common rice leaf diseases. The proposed model achieves the highest training accuracy of 99.78% and validation accuracy of 97.35%. The effectiveness of the proposed model is evaluated on a set of independent rice leaf disease images with the best accuracy of 97.82% with an area under curve (AUC) of 0.99. Besides that, binary classification experiments have been carried out and our proposed model achieves recognition rates of 97%, 96%, 96%, 93%, and 95% for Blast, Brownspot, Bacterial Leaf Blight, Sheath Blight and Tungro, respectively. These results demonstrate the effectiveness and superiority of our approach in comparison to the state-of-the-art CNN-based rice leaf disease recognition models.