Audiovisual calibration tools for examiners: the use of a “standardised student” to train undergraduate paediatric examiners

Abstract

Background: Our university recently changed to a new form of formative, in-course assessment for medical students in paediatrics, the mini-clinical evaluation exercise (mini-CEX). All examiners differ in their stringency (the "hawk/dove phenomenon"), and we saw this as an ideal opportunity to address interrater differences as well as train examiners in mini-CEX. Aims: To evaluate the usefulness of a training DVD at mini-CEX examiner training workshops, as a pilot for assessing the effect of feedback on the scoring tendencies of undergraduate paediatric examiners. Methods: Subjects were an opportunity sample of 27 paediatricians, paediatric registrars and staff grade paediatricians participating in 2 mini-CEX training workshops. They scored a DVD depicting a "standardised student" taking a paediatric history or examining a child at 3 levels of competency. Scores were charted to create a frequency histogram and the results fed back to examiners. The critique of the "student" was discussed and scores at the extremes explored with examiners. Results of the opportunity sample were compared to a "gold standard" score derived from the assessment of the same DVD clip by 5 experienced undergraduate paediatric examiners. Results: Interrater reliability was high for the "below expectations student" but low for the "meets expectations/borderline student" (MEBS), indicating wide variation in the expectations of paediatric clinicians examining at undergraduate level. Of a possible 24 marks (minimum 6), the median mark for the MEBS was 16 (range 10-20). Examiners were often unaware whether they were "hawks" or "doves" until scoring tendencies were explored in the training session. All workshop participants felt the training DVD helped them to better understand the mini-CEX process. The median of the experienced examiners was also 16, with a much narrower range (14-17) which we have previously reported. Conclusions: Examiners found the DVD scoring session helpful to familiarise themselves with mini-CEX and identify their rating style. Further work is in progress to objectively evaluate whether feedback about scoring tendency and comparison to a score agreed by experienced examiners will modify extreme scoring behaviour.