Deep Neural Networks (DNNs) are popular machine learning models that have gained popularity due to its good predictive accuracy and ability to automatically learn features from raw data. Convolutional Neural Networks (CNNs) are one such models that have gained popularity in the field of Computer Vision (CV). Despite the popularity, these models are notoriously black-box models. The decisions made by these models are not explainable. In this paper we propose a method to create a Neuron Activation Rate based Bipartite Graph (NARBG) , that can explain the decisions made by the model, based on the contributions of class specific features. In the proposed method, the features are extracted from the raw data using a CNN based architecture. From the extracted features, neuron activation rate is calculated. Based on these neuron activation rates, influential features for the target class prediction are identified. Then a bipartite graph named NARBG is trained using these influential features. The predictions of NARBG can be explained based on the features and the path in the graph that got activated for a given target class prediction. The proposed method performs on par with the other state-of-the-art methods in terms of accuracy.