Protein design energy functions have been developed over decades by leveraging physical forces approximation and knowledge-derived features. However, manual feature engineering and parameter tuning might suffer from knowledge bias. Learning potential energy functions fully from crystal structure data is promising to automatically discover unknown or highorder features contributing to the protein’s energy. Here we propose a novel data-driven energy-based model based on SE(3)-equivariant model for protein conformation, namely GraphEBM. By combining with the graph attention network, GraphEBM improve the massage passing on the chemical bond and capture the interatomic interaction and overlap. GraphEBM was benchmarked on the local rotamer recovery task and found to outperform both Rosetta and the state-of-the-art deep learning based methods. Furthermore, GraphEBM also yielded promising results on combinatorial side chain optimization, improving 13.8% X1 rotamer recovery to the Atom Transformer method on average.