In this paper, an algorithm is proposed to learn and evaluate different finite mixture models (FMMs) for data clustering using a new proposed criterion. The FMM corresponds to the minimum value of the proposed criterion is considered the most efficient FMM with compact and essential components for clustering an input data. The proposed algorithm is referred to as the EMCE algorithm in this paper. The selected FMM by the EMCE algorithm is efficient, in terms of its complexity and composed of compact and essential components. Essential components have minimum mutual information, that is, redundancy, among them, and therefore, they have minimum overlapping among them. The performance of the EMCE algorithm is compared with the performances of other algorithms in the literature. Results show the superiority of the proposed algorithm to other algorithms compared, especially with small data sets that are sparsely distributed or generated from overlapping clusters.