Assessing suitability for a PBL curriculum: evaluating a new student selection instrument.

Abstract

Context A new student selection instrument has been designed to assess candidate suitability for a problem-based learning, small group curriculum. Objective To evaluate the performance of the new teamwork selection instrument in terms of its discriminatory power, fairness, validity, reliability and acceptability among candidates. Sample A sample of 69 volunteer candidates attending for interview formed 13 teams of 5 or 6 candidates each. Each candidate was assessed independently by 2 assessors. Candidate performance in the exercise was used for instrument evaluation purposes only. Results The instrument demonstrated good item discrimination (item-total correlations r = between 0.75 and 0.83, P < 0.01); the potential for good agreement between raters (63% agreement, weighted kappa = 0.38, P < 0.01); strong internal consistency reliability (Cronbach's a = 0.93), and good acceptability among candidates. No sources of assessment bias were identified on the basis of candidates' age (univariate anova F = 0.43, P > 0.05), gender (unrelated samples t-test F = 1.2, P > 0.05) or socioeconomic background (univariate anova F = 0.85, P > 0.05). There was no statistically significant relationship between the candidates' performance in the new exercise and their performance in the standardised formal interview (r = - 0.37, P > 0.05); the instrument had limited predictive validity, and some of the measured attributes require conceptual clarification. Discussion Statistical and conceptual analysis highlights the scope for development in the teamwork exercise. The exercise appears to be well suited to assessing candidate suitability for a problem-based learning curriculum.