Ensemble classifiers using clustering have significantly improved classification and prediction accuracies of many systems. These types of ensemble approaches create multiple clusters to train the base classifiers. However, the problem with this is that each class might have many clusters and each cluster might have different number of samples, so an ensemble decision based on large number of clusters and different number of samples per class within a cluster produces biased and inaccurate results. Therefore, in this article, we propose a novel methodology to create an appropriate number of strong data clusters for each class and then balance them. Furthermore, an ensemble framework is proposed with base classifiers trained on strong and balanced data clusters. The proposed approach is implemented and evaluated on 24 benchmark data sets from the University of California Irvine (UCI) machine learning repository. An analysis of results using the proposed approach and the existing state-of-the-art ensemble classifier approaches is conducted and presented. A significance test is conducted to further validate the efficacy of the results and a detailed analysis is presented.