Hyperspectral images contain many bands that can be used to obtain object material information for object tracking and remote sensing. Nevertheless, neighboring bands of hyperspectral images are often highly correlated, and a large number of bands increase the complexity of model learning. This issue is worsened by the shortage of labeled hyperspectral videos for fine-tuning pre-trained deep neural networks. To tackle these challenges, this paper introduces a novel background-aware band selection method to model spatial changes of an object and its corresponding local region, which is capable of selecting discriminative bands for object representation while reducing computational complexity. Specifically, the object and local region of each band is compared with other bands to obtain their dissimilarity scores. Guided by these scores, the top three bands are selected and form a three-channel image. This image is then fed into an object tracker. Experimental results demonstrate the efficiency and effectiveness of the proposed method on a benchmark hyperspectral object tracking dataset.