Evaluation of a conversational AI system for simulated clinical scenarios in medical education: a mixed-methods multi-centre study

Effective consultation skills remain fundamental to high-quality healthcare delivery, influencing diagnostic accuracy, patient satisfaction, and clinical outcomes. This protocol describes a mixed-methods evaluation of conversational artificial intelligence (AI) for simulated clinical scenarios in undergraduate medical education. The study will recruit the entire third-year cohort of medical students (n=271) at the University of Bristol to compare AI virtual patients with traditional actor-based simulations across four standardised primary care scenarios. Quantitative assessment will employ validated measures including an abridged version of the Immersive Technology Evaluation Measure (ITEM) (see appendix A), the User Experience Questionnaire-Short (UEQ-S), and an adapted educational AI technology feedback questionnaire based on the Technology Acceptance Model (TAM), whilst qualitative data will be collected through focus groups analysed using Braun and Clarke's thematic framework. The primary outcome measures perceived realism, technical usability factors, and educational value of AI consultations compared to actor simulations. Secondary outcomes include AI’s comparative performance across (four) different clinical scenarios, feedback quality assessment, and recommendations for curricular integration. Statistical analysis will employ non-parametric methods with effect size calculations and confidence intervals, whilst qualitative analysis will be conducted independently by two researchers with subsequent consensus meetings. This comprehensive evaluation will provide evidence regarding the effectiveness of AI virtual patients for communication skills training, informing future integration into medical curricula whilst addressing stakeholder concerns about the role of technology in clinical education. Keywords: artificial intelligence, medical education, communication skills, virtual patients, simulation training, primary care, consultation skills