1083 Automated Generation and Human Evaluation of Neurosurgical Board Examination and Self-Assessment Questions
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
INTRODUCTION: Multiple-choice questions (MCQs) are the primary assessment format for neurosurgical board certification examinations. Creating high-quality examination questions requires significant expert time and resources in specialized fields like neurosurgery. METHODS: We developed an automated pipeline using OpenAI GPT-4o and Anthropic Claude Sonnet-3.5 to generate neurosurgical board-style questions from Neurosurgery Publications content. We generated 89,587 synthetic MCQs, 45,689 by GPT-4o and 43,898 by Claude. Each question was associated with a single image extracted from the articles’ figures. We evaluated the quality of synthetic questions through five surveys comparing 20 synthetic questions (10 from each model) with 10 authentic questions from the Self-Assessment for Neurological Surgeons (SANS) question bank. Each survey was completed by one neurosurgery resident and one attending neurosurgeon who assessed question source (human vs. AI) and suitability for board examination use. We also evaluated question-answering performance using GPT-4o and our specialized CNS-Obsidian model. RESULTS: SANS questions were more often perceived as human-made than GPT-generated (residents, p = 0.0002; attendings, p = 0.1091) and Claude-generated (residents, p = 0.0002; attendings, p = 0.0272) questions for both groups. Notably, 54% of AI-generated questions misled at least one evaluator, and 23% misled both. In quality assessments, SANS questions outperformed GPT-generated (residents, p < 10-5; attendings, p = 0.0001) and Claude-generated (residents and attendings, p < 10-5) questions. Overall, only 25% of AI-generated questions were rated as suitable for board examinations versus 72% of human-generated questions when measured by evaluator consensus (p < 10^-7). CONCLUSIONS: While quality gaps remain between AI-generated and human-created neurosurgical board examination questions, our approach demonstrates the potential of VLMs to augment assessment development in specialized medical fields, potentially reducing the burden on examination boards and credentialing organizations.
Fields of Study & Subjects
Artificial Intelligence in Healthcare and Education
Innovations in Medical Education
Artificial Intelligence in Education
Publication details
- Journal / source
- Neurosurgery
- Volume
- 72
- Issue
- Supplement_1
- Pages
- 200–200
- Publication date
- 26 March 2026
- Language
- EN