In this work, we learn a similarity measure that discriminates between inter-class and intra-class samples based on a statistical inference perspective. Where, a nonlinear combination of Mahalanobis is proposed to reflect the properties of a likelihood ratio test. Since an object appearance is influenced by the identity of the object and variations in the capturing process, we represent the feature vector, which is the difference between two samples in the differences space, as a sample that is drawn from a mixture of many distributions. This mixture consists of the identities distribution and other distributions of the variations in the capturing process, in case of dissimilar samples. However, in case of similar samples, the mixture consists of the variations in the capturing process distributions only. Using this representation the proposed similarity measure accurately discriminates between inter-class and intra-class samples. To highlight the good performance of the proposed similarity measure, it is tested on different computer vision applications: face verification and person re-identification. To illustrate how the proposed learning method can easily be used on large scale datasets, experiments are conducted on different challenging datasets: LFW, PubFig, ETHZ, and VIPeR. Moreover, in these experiments, we evaluate different stages e.g., features detector, descriptor type and descriptor dimension, which constitute the face verification pipeline. The experimental results confirm that our learning method outperforms the state-of-the-art.