Evaluation of a novel formative objective structured clinical examination for clinical reasoning processes in undergraduate medical education: a pilot study

PURPOSE: Current clinical reasoning assessments rely mostly on outcome-based rather than process-based indicators, which fails to account for early reasoning processes that impact overall diagnostic accuracy. Between 2022 and 2024, the National Board of Medical Examiners collaborated with medical school faculty members to develop a formative objective structured clinical examination for clinical reasoning (OSCE-CR), which used standardized patient encounters to assess observed student behaviors in three clinical reasoning process subdomains. This study evaluates the psychometric properties of scores from a large-scale pilot study to inform the utility of OSCE-CR as a formative assessment in undergraduate medical education. METHOD: The large-scale pilot study was conducted in fall 2023 and included 76 clerkship students across four different U.S. medical schools. All students completed four different standardized patient cases, and two trained faculty raters scored videos of each case-student combination on the three clinical reasoning process subdomains. Multivariate generalizability theory (G-theory) was used to evaluate score reliabilities (at the subdomain and composite levels), and generalized linear mixed modeling (GLMM) was used to evaluate the predictive utility of the subdomain scores on diagnostic accuracy. RESULTS: The multivariate G-theory results demonstrated acceptable subdomain (mean Φ coefficient = 0.466) and composite reliability estimates (mean Φ coefficient = 0.738). Reliability estimates were most influenced by number of cases, with the addition of more cases to the assessment yielding substantially higher reliability estimates, particularly at the subdomain level. The GLMM results indicated composite scores were a significant positive predictor of diagnostic accuracy (0.605; 95% CI, 0.053-1.157; P = .03). CONCLUSIONS: The results provide initial psychometric evidence for use of OSCE-CR to measure and provide feedback on relevant clinical reasoning processes linked to diagnostic accuracy. By moving beyond outcome-based indicators, OSCE-CR enables students to practice these process-related behaviors to improve their clinical reasoning and reduce future diagnostic errors.