Advances in single-cell sequencing techniques enable individual cells to be sequenced across multiple modalities simultaneously, such as transcriptomics, epigenomics, and proteomics. Integrating multi-omics single-cell data provides a deeper and more comprehensive vision of genomic mechanisms. Due to the huge distribution shift between modalities, current integration methods mostly align the modality through domain adaption or similar strategies. They achieved limited performance likely because the modalities are over-divergent. Here, we propose a novel single-cell multimodal fusion method, scFPN, to improve the learned embedding by a clustering strategy. Specifically, scFPN first embeds each modality data through a feature pyramid network with a modality-specific variational auto-encoder. The learned hierarchical embeddings are then fused and input into a dual self-supervision optimizing module for attracting similar cells and separating dissimilar cells. We conducted comprehensive experiments on recently produced six datasets from different sequencing platforms and demonstrated the superiority of scFPN over a variety of state-of-the-art methods. More importantly, scFPN showed bio-interpretability by marker enrichment analysis from de-noising and imputing raw profiles.