Evidencing improvement in examiner calibration in OSCEs

In developing and administering OSCE-type assessments, institutions can spend significant resources training examiners and designing stations/scoring instruments in attempts to ensure that they are well-calibrated. This paper, situated in the context of a high-stakes OSCE for international medical graduates wanting to work in the national health service in the UK, employs a recently developed quantitative measure of examiner calibration to identify which stations show relatively high and low average degrees of calibration between examiners. Using documentary analysis of station materials, we then investigate these stations qualitatively to better understand what factors might drive better calibration as stations develop and design elements change over time. We find that those stations that are better calibrated are typically newer with more detailed and relevant scoring guidance and support materials, whilst there is little evidence that the nature of the task(s) or other contextual factors are important in determining calibration levels. In this work, we provide strong evidence of how key developments in station design, the quality of support materials and enhanced examiner training practices can succeed in improving degrees of calibration in OSCE stations - and suggest ways that all institutions might improve their practices in this regard.