With great power comes great responsibility: How narrow conceptions of validity in high‐stakes testing undermine competence

CONTEXT: As Medical Education recognises the 100th anniversary of Kelley's writing on validity, ongoing efforts to implement competency-focused training make the centenary a critical time to reflect on the sufficiency of validity argumentation for high-stakes testing. Shifting dynamics in assessment, after all, have never been more apparent as recent years have witnessed countries that traditionally place heavy emphasis on high-stakes testing (e.g. the United States and Canada) respond to social pressures by decreasing their use while others (e.g. the UK) have moved the opposite direction. FOCUS: The needs of institutions, administrators, students, communities and patients are evolving within a context of dynamic changes in society, education and technology. We, therefore, offer critical analysis of three myths that continue to surround high-stakes testing as a means to encourage collective and deliberate reflection about how the field can best adapt. DISCUSSION: In the era of competency-based education, it is not only insufficient but also counter-productive to mount a validity argument without demonstrating how high-stakes decisions account for (and influence) what happens before and after moments of high-stakes testing. High-stakes decisions intended to assure competence can paradoxically, yet predictably, threaten competence if the assessments they rely on prioritise short-term hurdles over the skills and behaviours required for long-term success.