This paper presents a novel method for content-based color image retrieval that combines color vector quantization and visual primary features into a compact feature representation. Color vector quantization is proposed to describe the image in a compressed stream by preserving the contrast of an image and produce two color quantizers which are processed by vector quantization to preserve the content of a color image. Loosely inspired by human visual system and its mechanism in effectively recognizing objects in an image by its edges and color distribution, we propose extraction of visual primary features based on edge texture orientation and color moments features. The representation of the proposed method utilizes histogram-based features which are populated by color quantization and visual primary features that are used to measure the similarity between the two color images by a specific distance metric computation. The proposed method proves to be efficient and adaptive to the particulars of image retrieval, while it does not require any training information, making it suitable for real time color CBIR applications.The smaller feature size is an additional benefit of the proposed methodology which makes it ideal to use for embedded computing, mobile computing, energy efficient computing and high-throughput systems. An extensive experimental evaluation conducted on four publicly available datasets namely Wang, Vistex-640, Corel-5k and Corel-10k datasets against well-known fusion based, non-fusion based, and deep learning methods highlights the effectiveness and efficiency of the proposed method.