Background: This article is a guide to critically appraising reliability studies based upon fundamental reliability concepts. It focuses on reliability studies of physical measurement instruments used in the allied health professions. Content: Eight critical appraisal questions specific to reliability studies are outlined, and their theoretical basis and importance described. The questions encompass key aspects of reliability theory; selection of clinically stable participants appropriate to the instrument and condition of interest; minimization of random error; appropriate periods of time between measurements; the interpretation of frequently used reliability statistics; and the generalizability of results. The importance of interpreting reliability studies in specific clinical contexts and settings is emphasized. Conclusions: These questions will guide clinicians and researchers to make informed decisions regarding whether reliability evidence can be applied to their specific context.