1085 Automating Evaluation of Evidence and Recommendations in Congress of Neurological Surgeons Guideline Preparation Using a Large Language Model

View at publisher Summarise with VIRAT
Authors: Vijay Nitturi, Álex Flores, David F. Bauer
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

INTRODUCTION: The exponential growth of neurosurgical literature has made guideline creation an expansive and arduous process. Previously, we have shown the capability of untrained large language models (LLMs) to assist in abstract screening for Congress of Neurological Surgeons (CNS) guideline creation. Further work is necessary to determine if other parts of guideline creation can be automated. METHODS: Three guidelines were randomly selected from three different neurosurgical fields. Using the reference list, all full-text articles were downloaded; unavailable and ChatGPT-incompatible articles were removed. A standardized script for evidence grading was developed from the official CNS methodology and inputted into ChatGPT without access to the internet. ChatGPT grades for individual articles and overall recommendation levels were tabulated, and accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated with Level III status labeled as positive and non-Level III status as negative. RESULTS: Across 3 guidelines, 465 articles were downloaded, of which 15 were excluded. Ultimately, 423 Level III evidences and 27 Level I/II evidences were included in the analysis, with 48 overall recommendations. ChatGPT achieved an 81.6% accuracy, 81.3% sensitivity, 85.2% specificity, 98.9% PPV, and 22.5% NPV on grading individual articles. With overall recommendations, it achieved a 64.6% accuracy, 62.2% sensitivity, 100% specificity, 100% PPV, and 15.0% NPV. CONCLUSIONS: ChatGPT is a capable assistant in grading evidence and has an extremely high ability to detect Level III evidence appropriately. The process of appraising evidence in guideline preparation could be accelerated through LLM use. Future studies should aim to compare ChatGPT with other LLMs in evaluating evidence and auto-generating guideline recommendations.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Clinical practice guidelines implementation Meta-analysis and systematic reviews

Publication details

Journal / source
Neurosurgery
Volume
72
Issue
Supplement_1
Pages
201–201
Publication date
26 March 2026
Language
EN