Features are crucial for deep learning models as they help understand complex data, learn patterns, and make accurate predictions. Feature engineering can sometimes be computationally expensive, but it can speed up the training and inference phases of the deep learning models. By providing a more concise and informative representation, it reduces the number of parameters and computations required in the down-stream operations. Well-chosen features can enhance a model's ability to represent data in a structured and meaningful way. They help learn hierarchical dependencies in data, reduce the dimensionality of the input data, and generalize from the training data to unseen data to make the model robust. In this work, we present a self-supervised learning model for feature generation from RNA sequences towards applying in deep learning models for RNA contact map prediction. We test the efficacy of our extracted features by comparing the prediction performance with the prediction performance obtained by the features extracted using the state-of-the-art language foundation model, RNA-FM. We found our approach promising.