Research into non-technical skills has assisted in developing understanding of their use in commercial aviation. However, assessing these skills continues to present difficulties. This paper describes qualitative research involving five experienced airline check captains, to determine the criteria they use to assess pilots undergoing promotion to airline captain. The findings showed differential emphases on the criteria used during assessment. Flying skills, situational awareness and decision making were found to be essential, whilst aviation knowledge, management and communication assisted in maintaining these essential skills.