In their natural form, convolutional neural networks (CNNs) lack interpretability despite their effectiveness in visual categorization. Concept activation vectors (CAVs) offer human-interpretable quantitative explainability, utilizing feature maps from intermediate layers of CNNs. Current concept-based explainability methods assess explainer faithfulness primarily through Fidelity. However, relying solely on this metric has limitations. This study extends the Invertible Concept-based Explainer (ICE) to introduce a new ingredient measuring concept consistency. We propose the CoherentICE explainability framework for CNNs, expanding beyond Fidelity. Our analysis, for the first time, highlights that Coherence provides a more reliable faithfulness evaluation for CNNs, supported by empirical validations. Our findings emphasize that accurate concepts are meaningful only when consistently accurate and improve at deeper CNN layers.