Assessment of Large Language Model Performance on Virtual Patient Scenarios: Mixed Methods Study (Preprint)

BACKGROUND Generative AI is increasingly being explored in medical education as a tool to enhance clinical reasoning and support interactive learning. However, only a few studies have so far evaluated how such models perform in suitable educational settings. This study examines the effectiveness of ChatGPT in simulating clinical decision-making through virtual patient (VP) interactions. OBJECTIVE This study aims to evaluate and compare the accuracy of ChatGPT-3.5 and ChatGPT-4 in solving VP scenarios across medical specialties, and to explore their strengths, limitations, and educational implications. METHODS A total of 64 VP scenarios covering paediatric, adult, disease management, and oncology cases were tested using ChatGPT-3.5 and ChatGPT-4 within the MobiViP mobile platform. Responses were classified as correct, incorrect, or inadequate. Success rates were calculated using descriptive and inferential statistics. Inadequate responses were also analyzed thematically. RESULTS ChatGPT-4 significantly outperformed ChatGPT-3.5 across all categories (median success rate: 92.55% vs 78.68%, p