This paper proposes a novel heterogeneous specular and diffuse (HSD) 3-D surface approximation which considers spatial variability of specular and diffuse reflections in face modelling and recognition. Traditional 3-D face modelling and recognition methods constrain human faces with either the Lambertian assumption or the homogeneity assumption, resulting in suboptimal shape and texture models. The proposed HSD approach allows both specular and diffuse reflectance coefficients to vary spatially to better accommodate surface properties of real human faces. From a small number of face images of a person under different lighting conditions, 3-D shape and surface reflectivity property are estimated using a localized stochastic optimization method. The resultant personalized 3-D face model is used to render novel gallery views under different poses for recognition across pose. The proposed approach is evaluated on both synthetic and real face datasets and benchmarked against the state-of-the-art approaches. Experimental results demonstrated that it can achieve a higher level of performances in modelling accuracy, algorithm reliability, and recognition accuracy, which suggests that face modelling and recognition beyond the Lambertian and homogeneity assumptions is a feasible and better solution towards pose-invariant face recognition.