Peer Assessment of Aviation Performance: Inconsistent for Good Reasons
MetadataShow full item record
Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots (6 first officers, 6 captains, and 6 flight examiners). Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal (a) considerable variance within performance categories, (b) differences in the process used as evidence in support of a performance rating, (c) different numbers and types of facts (cues) identified, and (d) differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations.
© 2015 Cognitive Science Society. This is the pre-peer reviewed version of the following article: Peer Assessment of Aviation Performance: Inconsistent for Good Reasons, Cognitive Science, Vol.39 (2), 2015, pp.405–433,which has been published in final form at dx.doi.org/10.1111/cogs.12152.
Sensory Processes, Perception and Performance