Assessing the performance of commercial airline pilots has proven difficult, necessitating a closer look at how pilot performance is assessed in practice. The aim of this study is to explore the basis on which experienced captains assess their peers' safety-critical pilot performance. It is examined whether all assessors determine the same criteria, which for some assessors lead to unacceptable risk. Using the performance assessment grid implemented by their employer as a means for organizing their task, three pairs of airline captains assessed a captain and first officer in two video scenarios. The findings show that assessors use varying criteria to arrive at the same assessment or apply the same or very similar criteria to arrive at different assessments. Addressing certain criteria does not necessarily lead to the same assessment scores; rather, criteria receive different weights in the composition of the total score. Implications are drawn with respect to inter-rater reliability and the diversity of observations