1083 Automated Generation and Human Evaluation of Neurosurgical Board Examination and Self-Assessment Questions

View at publisher Summarise with VIRAT
Authors: Anton Alyakin, Shrutika Singh, Jaden Stryker, Daniel Alexander Alber, Jin Vivian Lee, Akshay Save, David B. Kurland, Cordelia Marcela Orillac, Aly Al-Amyn Valliani, Sean N. Neifert, Darryl Lau, Ilya Laufer
+6 more Peter A. Rozman, Eveline Teresa Hidalgo, Howard Antony Riina, Laura A. Snyder, Eric C. Leuthardt, Douglas S Kondziolka
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

INTRODUCTION: Multiple-choice questions (MCQs) are the primary assessment format for neurosurgical board certification examinations. Creating high-quality examination questions requires significant expert time and resources in specialized fields like neurosurgery. METHODS: We developed an automated pipeline using OpenAI GPT-4o and Anthropic Claude Sonnet-3.5 to generate neurosurgical board-style questions from Neurosurgery Publications content. We generated 89,587 synthetic MCQs, 45,689 by GPT-4o and 43,898 by Claude. Each question was associated with a single image extracted from the articles’ figures. We evaluated the quality of synthetic questions through five surveys comparing 20 synthetic questions (10 from each model) with 10 authentic questions from the Self-Assessment for Neurological Surgeons (SANS) question bank. Each survey was completed by one neurosurgery resident and one attending neurosurgeon who assessed question source (human vs. AI) and suitability for board examination use. We also evaluated question-answering performance using GPT-4o and our specialized CNS-Obsidian model. RESULTS: SANS questions were more often perceived as human-made than GPT-generated (residents, p = 0.0002; attendings, p = 0.1091) and Claude-generated (residents, p = 0.0002; attendings, p = 0.0272) questions for both groups. Notably, 54% of AI-generated questions misled at least one evaluator, and 23% misled both. In quality assessments, SANS questions outperformed GPT-generated (residents, p < 10-5; attendings, p = 0.0001) and Claude-generated (residents and attendings, p < 10-5) questions. Overall, only 25% of AI-generated questions were rated as suitable for board examinations versus 72% of human-generated questions when measured by evaluator consensus (p < 10^-7). CONCLUSIONS: While quality gaps remain between AI-generated and human-created neurosurgical board examination questions, our approach demonstrates the potential of VLMs to augment assessment development in specialized medical fields, potentially reducing the burden on examination boards and credentialing organizations.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Innovations in Medical Education Artificial Intelligence in Education

Publication details

Journal / source
Neurosurgery
Volume
72
Issue
Supplement_1
Pages
200–200
Publication date
26 March 2026
Language
EN