Understanding Variance in Pilot Performance Ratings: Two Studies of Flight Examiners, Captains, and First Officers Assessing the Performance of Peers

View/ Open
Author(s)
Mavin, Timothy
Roth, Michael
Dekker, Sidney
Year published
2013
Metadata
Show full item recordAbstract
Two studies were designed to investigate how pilots of different rank evaluate flight-deck performance. In each study, the pilots were asked to assess sets of three different videotaped scenarios featuring pilots in a simulator exhibiting poor, average, and good performance. Study 1, which included 92 airline pilots of differing rank, was aimed at comparing how individuals rate performance. The subjects used a standardized assessment form, which included six criteria, each having a 5-point rating scale. Analysis of the first study revealed that there was considerable variance in the performance ratings between flight ...
View more >Two studies were designed to investigate how pilots of different rank evaluate flight-deck performance. In each study, the pilots were asked to assess sets of three different videotaped scenarios featuring pilots in a simulator exhibiting poor, average, and good performance. Study 1, which included 92 airline pilots of differing rank, was aimed at comparing how individuals rate performance. The subjects used a standardized assessment form, which included six criteria, each having a 5-point rating scale. Analysis of the first study revealed that there was considerable variance in the performance ratings between flight examiners, captains, and first officers. The second study was designed to better understand the variance. Eighteen pilots (six flight examiners, six captains, and six first officers) working in pairs evaluated performances, in a modified think-aloud protocol. The results showed that there were good reasons for the observed variances. The results are discussed in relation to inter-rater reliability.
View less >
View more >Two studies were designed to investigate how pilots of different rank evaluate flight-deck performance. In each study, the pilots were asked to assess sets of three different videotaped scenarios featuring pilots in a simulator exhibiting poor, average, and good performance. Study 1, which included 92 airline pilots of differing rank, was aimed at comparing how individuals rate performance. The subjects used a standardized assessment form, which included six criteria, each having a 5-point rating scale. Analysis of the first study revealed that there was considerable variance in the performance ratings between flight examiners, captains, and first officers. The second study was designed to better understand the variance. Eighteen pilots (six flight examiners, six captains, and six first officers) working in pairs evaluated performances, in a modified think-aloud protocol. The results showed that there were good reasons for the observed variances. The results are discussed in relation to inter-rater reliability.
View less >
Journal Title
Aviation Psychology and Applied Human Factors
Volume
3
Issue
2
Copyright Statement
© 2013 Hogrefe Publishing. This is an electronic version of an article published in Aviation Psychology and Applied Human Factors, Volume 3, Issue 2, Pages 53-62, 2013 by Hogrefe Publishing. This article may not exactly replicate the final version published in Aviation Psychology and Applied Human Factors. It is not the version of record and is therefore not suitable for citation.
Subject
Technical, Further and Workplace Education
Vocational Education and Training Curriculum and Pedagogy
Teacher Education and Professional Development of Educators
Transportation and Freight Services
Psychology
Cognitive Sciences