This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication. Background Central line-associated bloodstream infections (CLABSI) are associated with significant morbidity and mortality. This condition is therefore the focus of quality initiatives, which primarily use audit and feedback to improve performance. However, feedback of quality data inconsistently affects clinician behavior. A hypothesis for this inconsistency is that a lack of comprehension of CLABSI data by decision makers prevents behavior change. In order to rigorously test this hypothesis, a comprehension scale is necessary. Therefore, we sought to develop a scale to assess comprehension of CLABSI quality metric data. Methods The initial instrument was constructed via an exploratory approach, including literature review and iterative item development. The developed instrument was administered to a sample of clinicians, and each item was scored dichotomously as correct or incorrect. Psychometric evaluation via exploratory factor analyses (using tetrachoric correlations) and Cronbach’s alpha were used to assess dimensionality and internal consistency. Results 97 clinicians responded and were included. Factor analyses yielded a scale with one factor containing four items with an eigenvalue of 2.55 and a Cronbach’s alpha of 0.82. The final solution was interpreted as an overall CLABSI “comprehension” scale given its unidimensionality and assessment of each piece of data within the CLABSI feedback report. The cohort had a mean performance on the scale of 49% correct (median = 50%). Conclusions We present the first psychometric evaluation of a preliminary scale that assesses clinician comprehension of CLABSI quality metric data. This scale has internal consistency, assesses clinically relevant concepts related to CLABSI comprehension, and is brief, which will assist in response rates. This scale has potential policy relevance as it could aid efforts to make quality metrics more effective in driving practice change.