Physicians Outperform Large Language Models in Pediatric Discharge Summary Generation

OBJECTIVE: Pediatric hospitalists manage increasing volumes of complex patients. Large language models (LLMs) may offer opportunities to reduce clinician workload through clinical documentation summarization. The objective of this study was to assess the quality of unedited LLM-generated discharge summaries compared with the quality of physician-authored discharge summaries. METHODS: Our study provided an anonymized, comparative evaluation of 35 unedited LLM-generated and 35 physician-authored discharge summaries graded by pediatric hospitalists and primary care pediatricians. Hospitalists used the validated Physician Documentation Quality Instrument (PDQI)-9, and primary care pediatricians used a shortened version of the instrument. Clinical Risk Group (CRG), length of stay, and primary documentation author training level were collected for each summary. Total and subdomain scores were compared along with the association of scores and clinical factors. RESULTS: Baseline encounter and documentation characteristics were similar between groups. LLM-generated discharge summaries were significantly longer than physician-authored discharge summaries (mean word count 403 vs 329, P < .001). Pediatric hospitalists rated the physician-authored summaries higher in overall score (27.4 vs 23.7, P < .001) and in all 9 PDQI subdomains. Primary care pediatricians rated physician-authored summaries higher in overall score (18.1 vs 15.6, P < .0001) and in 5 of 6 PDQI subdomains, with no significant difference in internal consistency. Spearman correlation showed an associated decrease in physician-authored score with increased CRG (ρ = -0.24, P = .01). CONCLUSIONS: Physicians outperformed LLMs in creating discharge summaries. Future studies should focus on the quality of physician-modified LLM-generated documentation and the effects on documentation quality, physician workload, and overall physician well-being.