Artificial Intelligence for Clinical Competency Assessment: A Scoping Review of Methods and Applications (Preprint)

BACKGROUND Strengthening the global health workforce is central to achieving Universal Health Coverage, yet existing approaches to measuring clinical competency remain resource-intensive, episodic, and difficult to scale, especially in low- and middle-income contexts. Recent advances in large language models (LLMs) have enabled AI-led simulated standardized patients (SSPs) that may offer scalable alternatives to traditional assessments. OBJECTIVE This study aims to systematically map and characterize the existing scope, design features, and validation approaches of AI-led SSP tools used for clinical competency assessment. METHODS We conducted a scoping review following JBI guidelines, searching MEDLINE, Embase, CINAHL, Education Source, and Web of Science from inception through June 2025. Two reviewers independently screened studies and extracted data across five domains: study characteristics and populations; frontend platform and interface features; backend AI models and architectures; user interaction and automatic feedback mechanisms; and tool evaluation methods and outcomes. RESULTS Between 2008 and 2025, 1,185 studies were identified and 21 studies met the inclusion criteria. Most described single-site pilot evaluations or prototype systems were developed within academic institutions in high-income countries, primarily targeting pre-licensure medical or nursing students. SSPs most commonly supported text-based, web-hosted history-taking, while simulations of physical examination, laboratory tests, diagnostic reasoning, and management planning were less common. Backend architectures relied heavily on human-authored case scripts and manually defined scoring criteria, with LLMs primarily enhancing conversational fluency rather than automating clinical reasoning or evaluation. Automated feedback and scoring were reported in approximately half of the studies and showed moderate-to-high agreement with human raters when evaluated, though validation evidence was heterogeneous and limited. CONCLUSIONS AI-led SSPs are emerging as accessible and realistic tools for clinical competency assessment, particularly across all levels of medical education. However, current implementations remain early-stage, human-dependent, and narrowly validated, constraining their widespread use as standardized or scalable instruments for health system workforce evaluation. Advancing SSPs toward end-to-end automated assessment tools will require integrated system designs, rigorous validation, and intentional development for deployment across diverse and resource-constrained settings.