Validation of Large Language Model-Based Extraction of NIH Stroke Scale Scores from Clinical Narratives: A Diagnostic Accuracy Study

The NIH Stroke Scale (NIHSS) is a standardized clinical instrument for quantifying neurological deficit severity in acute ischemic stroke. Developed in the late 1980s by the Stroke Research Group of the National Institutes of Health, the scale was designed to provide a rapid, objective, and reproducible bedside assessment. In the original validation study, Brott et al. (1989) demonstrated good to excellent interrater reliability, with weighted kappa values exceeding 0.60 for most individual items and high agreement for the total score. Subsequent studies consistently confirmed high interrater reliability across different professional groups and clinical settings, with intraclass correlation coefficients (ICC) for the total NIHSS score typically ranging from 0.82 to 0.95, particularly following standardized training (Lyden et al., 1994; Goldstein & Samsa, 1997; Dewey et al., 1999). These findings established the NIHSS as a reliable and reproducible severity measure and supported its widespread adoption in stroke trials, registries, and clinical guidelines. The objective of this study is to determine the validity and reliability of large language model–based extraction of NIHSS scores from routine clinical documentation. We hypothesize that LLM-derived NIHSS scores demonstrate high agreement with expert-rated reference standards and may serve as a scalable method for structured secondary use of unstructured stroke documentation. Primary objective To evaluate the overall agreement between NIHSS item scores extracted by a large language model (LLM) using two different prompt strategies (simple and refined prompts) and those assigned by two NIHSS-certified physicians serving as the reference standard, across all NIHSS items. The primary analysis will quantify overall item-level agreement between the LLM and the physician-derived reference ratings separately for each prompt strategy. Secondary study objective Item-specific agreement: To assess the agreement between the LLM and the physician reference standard for each individual NIHSS item, separately for the simple and refined prompt strategies. Interrater reliability between physician: To evaluate the agreement between the two NIHSS-certified physicians in order to quantify the human reference variability. Agreement of total NIHSS score: Agreement between the LLM-derived total NIHSS score and the physician-derived reference total NIHSS score, quantified using the intraclass correlation coefficient (ICC), separately for each prompt strategy. This comparison will be performed separately for each prompt strategy. Stratified analysis by stroke severity: To assess the primary endpoint (overall item-level agreement between LLM and physicians) stratified by stroke severity categories (mild, moderate, severe) and analyzed separately for both prompt strategies. Stroke severity categories will be defined according to established NIHSS thresholds (mild: 0–4, moderate: 5–14, severe: ≥15). Comparative performance of prompt strategies: To directly compare agreement metrics between the simple and refined prompt strategies to determine whether prompt refinement improves extraction accuracy.