Deep-learning models are popular machine learning models that have gained their popularity in various fields of computer vision, natural language processing etc, due to their excellent predictive accuracy and ability to automatically extract good features from raw input data. Though these models have these significant advantages, most deep learning models are notoriously black-box models. The features learned by these models are not explainable. The need to explain these predictions is important in many high stakes fields. In this work, we propose a method that uses convolutional layers like in Convolutional Neural Networks (CNNs) that automatically learns features from raw data. In our proposed method we extract feature representation that is unique to each target class in the given data. Visualizations of this unique representation explain the feature learned by the model for a specific target class. To classify an input data we use two approaches. The first approach uses the similarity scores to match the features learned for a particular instance with the extracted target class-specific features to make prediction. The second approach uses an intrinsically explainable model like logistic regression to make predictions. Thus our proposed method in addition to having good predictive accuracy, identifies features that are class-specific. The proposed method achieved an accuracy of 99.31 % on MNIST data set, 90.45% on CIFAR-10 data set and 96.45% accuracy on Assira data set.