A Large Language Model Approach to Functional Status Scale Assessment

OBJECTIVES: To develop a fine-tuned version of the generative pretrained transformer (GPT)-4o artificial intelligence (AI) model able to estimate Functional Status Scale (FSS) scores among critically ill children. DESIGN: Secondary analysis of a prospective, observational cohort of critically ill children 1 month to 18 years old who required invasive mechanical ventilation for greater than or equal to 3 days. Four patient notes from each of three hospitalization timepoints-baseline (history & physical), PICU transfer, and hospital discharge-along with retrospectively assigned FSS scores were used to train the model. The resulting custom GPT (hereafter, FSS-AI) was then applied to the remaining 428 notes. The GPT-generated FSS scores were then compared with the manually assigned scores determined prospectively during the original study. SETTING: A single, quaternary-care academic pediatric hospital. PATIENTS: Children who completed the original study, survived to discharge, and had FSS scores documented. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: FSS-AI analyzed 428 notes from 147 patients over 255 minutes (averaging 35.7 s/note). FSS-AI demonstrated moderate agreement with manually determined total FSS scores at the pre-illness baseline (weighted Cohen's Kappa, 0.59; 95% CI, 0.49-0.70) and hospital discharge (0.51; 95% CI, 0.43-0.58) timepoints, with slightly lower agreement at PICU transfer (0.45; 95% CI, 0.37-0.54). For discrimination of normal total FSS scores (6-7) from abnormal scores (≥ 8), FSS-AI accuracy and positive predictive value were highest at the pre-illness baseline (0.90 and 0.95, respectively) and hospital discharge (0.81-0.75) timepoints. FSS-AI identified children with a new morbidity at hospital discharge (total FSS increase ≥ 3 or domain FSS increase ≥ 2) with accuracy and sensitivity of 0.75 and 0.56. CONCLUSIONS: A custom version of GPT-4o was able to estimate FSS scores at multiple hospitalization timepoints. The tool demonstrated moderate agreement with manually determined scores, could discriminate children with normal vs. abnormal FSS (best performance at baseline and hospital discharge timepoints), and had fair accuracy for detecting new morbidities present at hospital discharge.