Validation of AI Platforms for Generation of Medical Education Assessment Questions: A Comparative Quality Evaluation Study Using Brain and Behavior Module Content

This study will evaluate and compare the ability of three widely available AI platforms — Google NotebookLM, ChatGPT, and Microsoft Copilot — to generate high-quality, board-style examination questions from preclinical medical education materials. Using lecture slides from the Brain & Behavior module at New York Medical College, each platform will be provided with identical, standardized prompts and asked to generate questions across five core topic areas: neurology basics, CNS structure and function, vascular neurology, neurodegenerative diseases, and epilepsy. The module encompasses 18 lectures totaling approximately 25 hours of curricular content, with an estimated 115–145 questions to be generated per platform. Generated questions will be evaluated by blinded faculty raters using a standardized rubric assessing accuracy, clinical relevance, and completeness. Inter-rater reliability will be quantified using Cohen's Kappa, and platform performance will be compared using the Kruskal-Wallis test. A reproducibility sub-study will assess the consistency of each platform's outputs across repeated identical prompts. We hypothesize that platforms will differ significantly in question quality, and that these differences will be detectable using blinded faculty evaluation.