Evaluating the Performance of Large Language Models in Vascular Surgery: A Case Series (Preprint)

BACKGROUND Large language models (LLMs) such as ChatGPT, Gemini and Claude are increasingly used by clinicians, yet their accuracy, safety and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning. OBJECTIVE This study aims to evaluate the performance and safety of three widely used LLMs in realistic vascular surgery scenarios to assess their fitness-for-use in clinical practice, and to elucidate the barriers that exist to their widespread integration. METHODS Forty-two fictitious vascular cases across five major pathologies: acute limb ischaemia (ALI), aortic disease (Ao), chronic limb ischaemic (CLI), diabetic foot infection (DFI) and extracranial cerebrovascular disease (ECD), were independently entered into ChatGPT-5, Gemini 2.5 and Claude Sonnet 4.5 using standardised prompts. Each model answered structured questions covering diagnosis, investigation and management (both operative and non-operative). Responses were scored by a panel /20 using a predefined rubric. Each plan’s overall safety was separately assessed. Comparative analyses utilised t-tests, ANOVA and multivariable logistic regression. RESULTS Mean composite scores by model were 86.5% (ChatGPT), 83.5% (Gemini), and 88.0% (Claude), whilst scores by disease were 82.7% (ALI), 83.8% (Ao), 85.9% (CLI), 86.1% (DFI) and 80.7% (ECD) (p=non-significant). Unsafe plans occurred in 11.9% (ChatGPT), 23.8% (Gemini) and 7.1% (Claude). On multivariable analysis, independent predictors of unsafe outputs were lower composite score (OR 0.47, p=0.001), higher word count (OR 1.003, p=0.001) and ALI (OR 20.9, p