In the past decade, linear representation based face recognition has become a very popular research subject in computer vision. This method assumes that faces belonging to one individual reside in a low-dimensional linear subspace. In real-world applications, however, face images usually are of degraded quality due to expression variations, disguises, and partial occlusions. These problems undermine the validity of the subspace assumption and thus the recognition performance deteriorates significantly. In this work, we propose a simple yet effective framework to address the problem. Observing that the linear subspace assumption is more reliable on certain face patches rather than on the holistic face, Probabilistic Patch Representations (PPRs) are randomly generated, according to the Bayesian theory. We then train an ensemble model over the patch-representations by minimizing the empirical risk w.r.t. the “leave-one-out margins”, which we term Linear Representation Ensemble (LRE). In the test stage, to handle the non-facial or novel face patterns, we design a simple inference method to dynamically tune the ensemble weights according to the proposed Generic Face Confidence (GFC). Furthermore, to accommodate immense PPR sets, a boosting-like algorithm is also derived. In addition, we theoretically prove two desirable property of the proposed learning methods. We extensively evaluate the proposed methods on four public face dataset, i.e., Yale-B, AR, FRGC and LFW, and the results demonstrate the superiority of both our two methods over many other state-of-the art algorithms, in terms of both recognition accuracy and computational efficiency.