The Career Decision Self-Efficacy Scale-Short Form (CDSE-SF) is one of the most frequently used instruments to assess individual levels of career-related self-efficacy. This study used the partial credit model within the framework of item response theory to examine the content, structural, substantive, and generalizability aspects of validity for the CDSE-SF in a sample of 534 Australian high school students aged between 14 and 19 years. The results showed clear evidence of multidimensionality for the CDSE-SF. Furthermore, there was strong support for the content, structural, and substantive aspects of validity when using the five subscales individually. The evidence of measurement invariance was found across grade levels; however, there were individual items that exhibited differential item functioning across gender, achievement level, and age-groups. The implications for career counseling and research are discussed.