The Transformer is a deep neural network based on the self-attention mechanism, designed to handle sequential data. Given its tremendous advantages in natural language processing, it has gained traction for other applications. As the primary structure of RNA is a sequence of nucleotides, researchers have applied Transformers to predict secondary and tertiary structures from RNA sequences. The number of Transformer-based models in structure prediction tasks is rapidly increasing as they have performed on par or better than other deep learning networks, such as Convolutional and Recurrent Neural Networks. This article thoroughly examines Transformer-based RNA structure prediction models. Through an in-depth analysis of the models, we aim to explain how their architectural innovations improve their performances and what they still lack. As Transformer-based techniques for RNA structure prediction continue to evolve, this review serves as both a record of past achievements and a guide for future avenues.