Investigating Assessors’ Scoring and Reasoning of Safety-Critical Crew Performance and Collaboration

Abstract

The use of an assessment model has shown large variation (disagreement) in the scoring of airline professionals who judged the performance of a captain and first officer in multiple video scenarios. A better understanding was thus required of the source of assessors’ disagreement in terms of both the scoring and reasoning of safety-critical crew performance and collaboration. In the first study, the present thesis quantitatively compared the scoring and assessment time of airline professionals (first officers, captains, and flight examiners) from two different airlines who were given versus not given an assessment model to judge performance. The demographics of the participants were compared in terms of their age, total flight hours, and years flown as commercial pilot. In contrast, the second and third studies qualitatively investigated and analysed assessors’ reasoning further. The second study closely examined the reasoning of captain assessor pairs who assessed the performance of a captain in two video scenarios. Individually for each assessor pair and pilot assessed, this study contrasted the reasoning of the assessors against the scores they provided to the pilot assessed. The third study looked into the reasoning of flight examiner assessor pairs who judged a captain and first officer performing in an engine fire scenario. It was examined whether commonly used assessment categories (e.g., Situational Awareness, Management) are specific, in the sense that assessors build their assessments on the same observations and reasoning (justifications). Furthermore, it was questioned what might distinguish the sort of observations that are specific to a single assessment category from observations that are stated in multiple categories.