Current assessment practice and its models typically aspire to capture the considerable variety of airline pilots' performance across highly specific behavioural markers and highly unspecific categories. Yet research into one of the most advanced assessment models has shown large disagreement among assessors in terms of both their scoring and reasoning. Rather than further attempts to eliminate this variety in assessors' judgments, a deeper understanding is required of the source of disagreement. The aim of this study is to reflect on current assessment practice by considering opposing, yet complementary views of human error and safety. Considerable discrepancy is identified between the underlying assumptions of assessment practice and the precepts of advanced safety research. The shift that has occurred in safety thinking hardly seems to have penetrated assessment practice in aviation. This study challenges current assessment practice and sketches approaches that might be more congruent with insight from safety research.