Image variability that is impossible or difficult to restore by intra-image processing, such as the variability caused by occlusions, significantly reduces the performance of image-recognition methods. To address this issue, we propose that the pixels associated with large distances obtained by inter-image pixel-by-pixels comparisons should be considered as inter-image outliers and should be removed from the similarity calculation used for the image classification. When this method is combined with the template-matching method for image recognition, it leads to state-of-the-art recognition performance: 91% with AR database that includes occluded face images, 90% with PUT database that includes pose variations of face images and 100% with EYale B database that includes images with large illumination variation