Large language models enhance diagnostic reasoning of medical students in rheumatology: a randomized controlled trial

BACKGROUND: Diagnostic errors and delays are common in rheumatology, driven by overlapping symptoms and the rarity of many diseases. While traditional diagnostic decision support systems (DDSS) have seen limited adoption because of high input burden and low perceived value, large language models (LLMs) now offer genuine dialogue and reduced effort, with rapidly improving diagnostic performance, yet empirical evidence on their real-world effectiveness and educational impact is still scarce. OBJECTIVE: The aim of this study was to investigate the impact of an LLM on medical students’ diagnostic performance in rheumatology compared with traditional resources. METHODS: In this randomized controlled trial, medical students solved three rheumatology vignettes. For each case, they provided a main diagnosis with confidence and up to four differential diagnoses. Participants were randomized to use ChatGPT-4o plus traditional resources or traditional resources alone. The primary outcome was the proportion of correct top diagnoses. Secondary outcomes were correctness within the top 5 diagnoses, a cumulative diagnostic score, diagnostic confidence, and completion time. RESULTS: Sixty-eight students (mean [SD] age 24.8 [2.6] years) were randomized. The LLM group identified the correct top diagnosis more often than controls (77.5% vs. 32.4%), yielding an adjusted odds ratio of 7.0 (95% CI 3.8–14.4; P