Assessing the Accuracy of ChatGPT Responses to Patient Questions in Pediatric Neurology: An Observational Study (Preprint)

BACKGROUND ChatGPT has seen increased use by caregivers as a source of medical information. Pediatric neurology has added complexities because there are various disorders, changing diagnoses and potential consequences of misinformation. OBJECTIVE To evaluate the accuracy, comprehensiveness and reproducibility of ChatGPT responses to frequently asked pediatric neurology questions. METHODS In total, 105 unique questions across seven pediatric neurology subspecialties were included. Every question was input twice into ChatGPT (GPT-5) during separate sessions. Two certified pediatric neurologists reviewed the responses and rated them on a 4-point scale: 1-complete and accurate,2- correct but not complete,3- partially incorrect,4-not correct. Reproducibility was considered to be agreement between grading categories (grade 1-2 and grade 3-4) throughout repeated responses. RESULTS 60.95%(64/105) of responses were rated as comprehensive, while 31.43%(33/105) were correct but incomplete; fully incorrect responses were uncommon 3.8%(4/105). Accuracy and reproducibility were highest for structured conditions such as cerebral palsy and movement disorders, and lower for more heterogeneous domains including neurodevelopmental delay and epilepsy. Overall reproducibility was high 95.2%(100/105). CONCLUSIONS ChatGPT mostly delivers accurate information in pediatric neurology, with variable depth of response depending on the disorder. It is most appropriate for caregiver education with oversight by the clinician. CLINICALTRIAL not applicable