Feature-level fusion approaches for multispectral biometrics are mainly grouped into two categories: 1) concatenation and 2) elementwise multiplication. While concatenation of feature vectors has benefits in allowing all elements to interact, it is difficult to learn output classification. Differently, elementwise multiplication has the benefits in enabling multiplicative interaction, but it is difficult to learn input embedding. In this paper, we propose a novel approach to combine the benefits of both categories based on a compact representation of two feature vectors' outer product, which is called the multimodal compact multi-linear pooling technique. We first propose to expand the bilinear pooling technique for two inputs to a multi-linear technique to accommodate for multiple inputs (multiple inputs from multiple spectra are frequent in the multispectral biometric context). This fusion approach not only allows all elements to interact and enables multiplicative interaction, but also uses a small number of parameters and low computation complexity. Based on this fusion proposal, we subsequently propose a complete multispectral periocular recognition system. Employing higher order spectra features with an elliptical sampling approach proposed by Algashaam et al., our proposed system achieves the state-of-the-art performance in both our own and the IIIT multispectral periocular data sets. The proposed approach can also be extended to other biometric modalities.