Investigating Assessors’ Scoring and Reasoning of Safety-Critical Crew Performance and Collaboration

View/ Open
Author(s)
Primary Supervisor
Dekker, Sidney
Sang, Robert
Other Supervisors
Mavin, Tim
Roth, Wolff-Michael
Year published
2015
Metadata
Show full item recordAbstract
The use of an assessment model has shown large variation (disagreement) in the scoring of airline professionals who judged the performance of a captain and first officer in multiple video scenarios. A better understanding was thus required of the source of assessors’ disagreement in terms of both the scoring and reasoning of safety-critical crew performance and collaboration.
In the first study, the present thesis quantitatively compared the scoring and assessment time of airline professionals (first officers, captains, and flight examiners) from two different airlines who were given versus not given an assessment model ...
View more >The use of an assessment model has shown large variation (disagreement) in the scoring of airline professionals who judged the performance of a captain and first officer in multiple video scenarios. A better understanding was thus required of the source of assessors’ disagreement in terms of both the scoring and reasoning of safety-critical crew performance and collaboration. In the first study, the present thesis quantitatively compared the scoring and assessment time of airline professionals (first officers, captains, and flight examiners) from two different airlines who were given versus not given an assessment model to judge performance. The demographics of the participants were compared in terms of their age, total flight hours, and years flown as commercial pilot. In contrast, the second and third studies qualitatively investigated and analysed assessors’ reasoning further. The second study closely examined the reasoning of captain assessor pairs who assessed the performance of a captain in two video scenarios. Individually for each assessor pair and pilot assessed, this study contrasted the reasoning of the assessors against the scores they provided to the pilot assessed. The third study looked into the reasoning of flight examiner assessor pairs who judged a captain and first officer performing in an engine fire scenario. It was examined whether commonly used assessment categories (e.g., Situational Awareness, Management) are specific, in the sense that assessors build their assessments on the same observations and reasoning (justifications). Furthermore, it was questioned what might distinguish the sort of observations that are specific to a single assessment category from observations that are stated in multiple categories.
View less >
View more >The use of an assessment model has shown large variation (disagreement) in the scoring of airline professionals who judged the performance of a captain and first officer in multiple video scenarios. A better understanding was thus required of the source of assessors’ disagreement in terms of both the scoring and reasoning of safety-critical crew performance and collaboration. In the first study, the present thesis quantitatively compared the scoring and assessment time of airline professionals (first officers, captains, and flight examiners) from two different airlines who were given versus not given an assessment model to judge performance. The demographics of the participants were compared in terms of their age, total flight hours, and years flown as commercial pilot. In contrast, the second and third studies qualitatively investigated and analysed assessors’ reasoning further. The second study closely examined the reasoning of captain assessor pairs who assessed the performance of a captain in two video scenarios. Individually for each assessor pair and pilot assessed, this study contrasted the reasoning of the assessors against the scores they provided to the pilot assessed. The third study looked into the reasoning of flight examiner assessor pairs who judged a captain and first officer performing in an engine fire scenario. It was examined whether commonly used assessment categories (e.g., Situational Awareness, Management) are specific, in the sense that assessors build their assessments on the same observations and reasoning (justifications). Furthermore, it was questioned what might distinguish the sort of observations that are specific to a single assessment category from observations that are stated in multiple categories.
View less >
Thesis Type
Thesis (PhD Doctorate)
Degree Program
Doctor of Philosophy (PhD)
School
School of Natural Sciences
Copyright Statement
The author owns the copyright in this thesis, unless stated otherwise.
Item Access Status
Public
Note
In order to comply with copyright the articles in chapters, 4, 5, 6 and 7 have not been published.
Subject
Pilot assessment
Airline professionals assessment
Assessment models