Facial image-based gender classification has been widely used in many real-world applications. Most of the existing work, however, focuses on designing sophisticated or specific feature descriptors for the entire face, which neglects the discriminative information carried by facial components and pattern order combinations. To address the issue, in this paper, we first propose a generalized texture operator, i.e., the multi-spatial multi-order interlaced pattern (MMIP) matrix, to represent the gender information possessed by the facial subregions with textural pattern orders. A chain-type support vector machine (CSVM) based feature vector selection scheme, is then developed to highlight the gender characteristics. As a result, the discriminative subregions and pattern orders are constructed as the feature representation for facial gender classification. We evaluate our proposed method on four benchmark datasets (i.e., FRGC 2.0, FERET, LFW and UND) for gender classification and demonstrate its interpretability, effectiveness and efficiency compared with state-of-the-art methods.