Background: We aimed to determine the psychometric properties and factor structure of the 19-item Female Sexual Function Index (FSFI) in 132 sexually active women previously treated for breast cancer. Methods: Confirmatory factor analysis explored three models: (a) second-order six-factor, (b) six-factor, and (c) five-factor models combining the desire and arousal subscales. Results: Results revealed excellent reliability for the total score (Cronbach's α = 0.94), and domain scores (all Cronbach's αs > 0.90), and good convergent and discriminant validity. The six-factor model provided the best fit of the models assessed, but a marginal overall fit (Tucker–Lewis index = 0.91, comparative fit index = 0.93, root mean square error of approximation = 0.09). Exploratory factor analyses (EFA) supported a four-factor structure, revealing an arousal/orgasm factor alongside the original pain, lubrication, and satisfaction domains. Conclusion: The arousal/orgasm factor suggests a “sexual response” construct, potentially arising from an underlying latent factor involving physical and mental stimulation in conceptualizations of arousal and orgasm in women treated for breast cancer. Finally, the EFA failed to capture an underlying desire factor, potentially due to measurement error associated with the small number of items (two) in this domain. Despite evidence that the FSFI has sound psychometric properties, our results suggest that the current conceptualizations of the FSFI might not accurately represent sexual functioning in women previously treated for breast cancer. Further research is required to elucidate the factors that influence desire, arousal, and orgasm in sexually active women in this population, and the reasons underlying sexual inactivity. Practical and theoretical implications for FSFI use in this population are discussed.