1085 Automating Evaluation of Evidence and Recommendations in Congress of Neurological Surgeons Guideline Preparation Using a Large Language Model
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
INTRODUCTION: The exponential growth of neurosurgical literature has made guideline creation an expansive and arduous process. Previously, we have shown the capability of untrained large language models (LLMs) to assist in abstract screening for Congress of Neurological Surgeons (CNS) guideline creation. Further work is necessary to determine if other parts of guideline creation can be automated. METHODS: Three guidelines were randomly selected from three different neurosurgical fields. Using the reference list, all full-text articles were downloaded; unavailable and ChatGPT-incompatible articles were removed. A standardized script for evidence grading was developed from the official CNS methodology and inputted into ChatGPT without access to the internet. ChatGPT grades for individual articles and overall recommendation levels were tabulated, and accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated with Level III status labeled as positive and non-Level III status as negative. RESULTS: Across 3 guidelines, 465 articles were downloaded, of which 15 were excluded. Ultimately, 423 Level III evidences and 27 Level I/II evidences were included in the analysis, with 48 overall recommendations. ChatGPT achieved an 81.6% accuracy, 81.3% sensitivity, 85.2% specificity, 98.9% PPV, and 22.5% NPV on grading individual articles. With overall recommendations, it achieved a 64.6% accuracy, 62.2% sensitivity, 100% specificity, 100% PPV, and 15.0% NPV. CONCLUSIONS: ChatGPT is a capable assistant in grading evidence and has an extremely high ability to detect Level III evidence appropriately. The process of appraising evidence in guideline preparation could be accelerated through LLM use. Future studies should aim to compare ChatGPT with other LLMs in evaluating evidence and auto-generating guideline recommendations.
Fields of Study & Subjects
Artificial Intelligence in Healthcare and Education
Clinical practice guidelines implementation
Meta-analysis and systematic reviews
Publication details
- Journal / source
- Neurosurgery
- Volume
- 72
- Issue
- Supplement_1
- Pages
- 201–201
- Publication date
- 26 March 2026
- Language
- EN