2109 Development and Validation of a Retrieval-Augmented Large Language Model for Neurosurgical Education

View at publisher Summarise with VIRAT
Authors: Sameer C. Sundrani, Ranbir Ahluwalia, Iyan Younus, Soren Jonzzon, Naveed Kamal, Breanne E. Reisen, Robert J. Dambrino, Harsh Jain, Lola Blackwell Chambless
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

INTRODUCTION: Neurosurgical education requires assimilation of knowledge from diverse resources. Traditional methods of information retrieval can be time-consuming and may not always provide context-specific answers to clinical questions. Decision support tools leveraging artificial intelligence offer potential solutions to these challenges. METHODS: A LLM-based tool was developed incorporating RAG architecture based on commonly referenced neurosurgery literature. The system was designed to provide evidence-based responses and to decline answering should the context not provide the needed information. Validation was conducted in three phases: (1) expert-assessment with inter-rater reliability assessed with Gwet's AC1 across 6 resident neurosurgeons, blinded to model architecture and other reviewers' responses, evaluating accuracy and reliability on 76 common neurosurgical questions spanning 10 categories, (2) pressure testing with 100 random out-of-context questions for safety assessment, and (3) testing against over 9,000 synthetic clinical vignettes to evaluate range of clinical utility for learners. RESULTS: The tool demonstrated 80.6% accuracy (95% CI: 76.7-84.0%) and 91.0% relevance (95% CI: 88.0-93.3%) on common neurosurgical questions. Performance varied significantly by domain (χ2 = 37.61, p = 2.0e-5), with highest accuracies in Spine (96.2%) and Vascular (89.6%) and lowest in Neuroradiology (62.3%). There was substantial inter-rater reliability for accuracy (Gwet's AC1 = 0.68), and relevance demonstrated almost perfect alignment (AC1 = 0.83). During pressure testing with out-of-context questions, the system appropriately declined to answer in 100% of cases. Finally, the system answered 76.5% (7,160/9,362) of queries from synthetic neurosurgical vignettes. CONCLUSIONS: This educational LLM tool represents a highly efficient and scalable approach to accessing neurosurgical knowledge. While not intended to replace clinical judgment or formal education, LLMs represent an opportunity to integrate real-time learning for trainees.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Information Retrieval and Search Behavior Machine Learning in Healthcare

Publication details

Journal / source
Neurosurgery
Volume
72
Issue
Supplement_1
Pages
274–274
Publication date
26 March 2026
Language
EN