Robust Learning Based on The Information Theoretic Learning

Abstract

Robust learning plays an important role in many fields such as computer vision, machine learning, pattern recognition, image processing, signal processing, etc. Especially, subspace learning and sparse learning (SLSL) have been widely used as useful tools to extract features in highly redundant raw data. However, most existing SLSL algorithms establish their learning models based on a second order statistic measurement, e.g. mean square error, which is sensitive to heavy noise and outliers/ occlusions in the data. This thesis devises advanced algorithms for robust SLSL for dimensional reduction of higher order tensor data and for learning sparse coefficients in the presence of outliers/occlusions. In this thesis, we concentrate on formulating new mathematical models for SLSL to solve outlier sample and sample outlier problems based on the information theoretic learning (ITL). For robust subspace learning, we developed two algorithms based on ITL-based metrics. We first take the advantages of MCC in suppressing outlier information to solve two-dimensional singular value decomposition in the presence of outlier samples and sample outliers, and then extend the framework to higher order tensor decomposition. A half-quadratic iterative optimization method is proposed to solve the proposed objective function. The proposed algorithm achieves better performance in face image reconstruction and classification. However, the MCC-based loss function uses the second order measurement to constraint the representation error, which is not always the best choice. To solve this problem, we then propose the generalized correntropy criterion to give more flexibility in controlling the reconstruction error. The experimental results in face image reconstruction, classification, and clustering show the advantages of the proposed algorithm. For robust sparse learning, we develop two robust orthogonal matching pursuit algorithms to learn robust sparse features based on ITL. Since the correntropy and generalized correntropy are both based on the second order statistics in the kernel space, which still will be influenced by outliers, we proposed a kernel non-second order statistics measurement in the kernel space for robust sparse learning in orthogonal matching pursuit. Besides, in the original matching pursuit algorithm, the correlation between two vectors are measured by the inner product operation. However, the inner product operation is not a robust function, which will magnify the effect from outliers. Based on this, a new correlation function is proposed based on the ITL to make the atom selection procedure more accurate.