Hyperspectral object tracking is notably challenging due to the high-dimensional nature of the data and the necessity of seamlessly integrating spectral, spatial and temporal information. Traditional methods often emphasize detection-based or tracking-based networks, each leveraging their inherent strengths but overlooking the potential advantages of a combined approach, leading to suboptimal performance in complex, real-world scenarios. Furthermore, this challenge is amplified by the variability of spectral bands across datasets, making the maintenance of consistent tracking performance complicated. To address these issues, we propose a novel, unified approach that merges adaptive band selection with a multi-model ensemble strategy. We introduce a local and global attention-based unified band selection (UBS) technique that identifies the most informative three bands from any dataset, significantly reducing data complexity while preserving critical spectral and spatial information. This UBS method employs spectral independence, allowing it to process hyperspectral video frames with any number of bands as input, ultimately generating a three-band pseudocolor image. This is coupled with a multi-model ensemble framework, utilizing a local and global attention-based appearance module. The module selects the optimal candidate by computing the similarity between the proposals generated by the base models and historical frames. Experimental results show that our approach, UBSTrack, achieves state-of-the-art performance, delivering robust and accurate tracking under different real-world challenging scenarios. The code of UBSTrack is available at the following link: source code.