Hyperspectral unmixing is a crucial task for identifying the constituent materials and their respective distributions in a scene. Utilizing known spectral libraries as prior information, semi-blind unmixing methods (also known as spectral-library-based methods) have been proven advantageous over unblind and blind approaches. However, such methods encounter two main challenges: difficulty in handling large-scale spectral libraries and vulnerability to variabilities stemming from differences between the underlying signatures and those in the spectral library. To address these challenges, a novel approach named DiffUn, based on a diffusion model, is proposed in this article for semi-blind hyperspectral unmixing. DiffUn considers hyperspectral unmixing as a sampling process from a posterior distribution, where the prior distribution is learned from a spectral library, and the likelihood distribution is estimated from the observed data by the linear spectral mixture model. Specifically, the approach first learns the spectral prior distribution from a spectral library through an unconditional diffusion model, then integrates this prior knowledge into the reverse process of the diffusion model, and finally samples the underlying endmembers and corresponding abundances from the posterior distribution. Since spectral prior distribution estimation is not sensitive to library size, DiffUn exhibits superior unmixing performance even in a large-scale library. Furthermore, DiffUn permits sampling spectral signatures from a continuous probabilistic distribution, whereas conventional semi-blind unmixing methods only allow endmembers selected from the library, which is a discrete space. Thus, DiffUn shows greater robustness to spectral variations. Experimental results on synthetic and real-world datasets demonstrate DiffUn outperforming the state-of-the-art semi-blind unmixing methods. The code is available at https://github.com/Dmsw/DiffUn.git.