Two studies were designed to investigate how pilots of different rank evaluate flight-deck performances shown in videotaped scenarios. Study one consisted of 92 airline pilots of differing rank. The second study involved eighteen pilots (six flight examiners, six captains, and six first offices) evaluating performances in pairs of similar-ranked pilot. Both studies asked subjects to examine 3 different videos featuring pilots in a simulator exhibiting poor, average, and good performance. The subjects used a standardized assessment form, which included six measures, each having a 5-point rating scale. Study 1 had pilots assess individually, where study two asked paired pilots to collectively assess the performance of pilots using a single assessment form. Preliminary analyses of the first study show that there are great variations in the performance ratings within flight examiners, captains, and first officers. These variations are taken up in the second study, which reveals reasons for why the pilots of all ranks arrived at very different assessments. The results are interpreted with respect to the potential of improving pilots' reflections on practice.