Human–AI collaboration in clinical reasoning: a UK replication and interaction analysis

OBJECTIVES: A paper from Goh et al. found that a large language model (LLM) working alone outperformed American clinicians assisted by the same LLM in diagnostic reasoning tests (Goh E, Gallo R, Hom J, Strong E, Weng J, Kerman H, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open 2024;7:e2440969). We aimed to replicate this experiment in a UK setting and explore how interactions with the LLM might explain the observed gaps in performance. METHODS: This was a within-subjects study of UK physicians. 22 participants answered structured questions on four clinical vignettes. For 2 cases physicians had access to an LLM via a custom-built web-application. Results were analysed using a mixed-effects model accounting for case difficulty and the variability of clinicians at baseline. Qualitative analysis involved coding of participant-LLM interaction logs and evaluating the rates of LLM use per question. RESULTS: Physicians with LLM assistance scored significantly lower than the LLM alone (mean difference 21.3 percentage points, p