The importance of the rare microbial biosphere in maintaining biodiversity and ecological functions has been highlighted recently. However, the current understanding of the spatial distribution of rare microbial taxa is still limited, with only a few investigations for rare prokaryotes and virtually none for rare fungi. Here, we investigated the spatial patterns of rare and abundant fungal taxa in alpine grassland soils across 2,000 km of the Qinghai-Tibetan plateau. We found that most locally rare fungal taxa remained rare (13.07%) or were absent (82.85%) in other sites, whereas only a small proportion (4.06%) shifted between rare and abundant among sites. Although they differed in terms of diversity levels and compositions, the distance decay relationships of both the rare and the abundant fungal taxa were valid and displayed similar turnover rates. Moreover, the community assemblies of both rare and abundant fungal taxa were predominantly controlled by deterministic rather than stochastic processes. Notably, the community composition of rare rather than abundant fungal taxa associated with the plant community composition. In summary, this study advances our understanding of the biogeographic features of rare fungal taxa in alpine grasslands and highlights the concordance between plant communities and rare fungal subcommunities in soil. IMPORTANCE Our current understanding of the ecology and functions of rare microbial taxa largely relies on research conducted on prokaryotes. Despite the key ecological roles of soil fungi, little is known about the biogeographic patterns and drivers of rare and abundant fungi in soils. In this study, we investigated the spatial patterns of rare and abundant fungal taxa in Qinghai-Tibetan plateau (QTP) alpine grassland soils across 2,000 km, with a special concentration on the importance of the plant communities in shaping rare fungal taxa. We showed that rare fungal taxa generally had a biogeographic pattern that was similar to that of abundant fungal taxa in alpine grassland soils on the QTP. Furthermore, the plant community composition was strongly related to the community composition of rare taxa but not abundant taxa. In summary, this study significantly increases our biogeographic and ecological knowledge of rare fungal taxa in alpine grassland soils.