Performance variation and implementation barriers of large language models in clinical healthcare: a systematic review

Abstract Large language models (LLMs) are a rapidly evolving class of artificial intelligence with significant potential in clinical healthcare. Despite accelerating adoption, rigorous systematic evidence on clinical utility, patient safety, and implementation feasibility remains fragmented. To systematically review LLM applications across clinical domains, evaluate performance with appropriate contextual caveats, characterize implementation barriers, and identify ethical and regulatory considerations. Scientific databases were searched from January 2020 to January 2025. Studies evaluating transformer-based LLMs (≥10M parameters) in clinical settings were eligible. Data were independently double-extracted; quality was assessed using QUADAS-2, RE-AIM, and TRIPOD frameworks. Due to substantial heterogeneity across domains, narrative synthesis was conducted per SWiM guidelines; descriptive statistics are presented for the one sufficiently homogeneous domain (clinical documentation, domain-adapted models, n=12). Fifty-two studies were included. Domain-adapted models (ClinicalBERT, BioBERT, Llama-3-8B) outperformed general-purpose models (GPT-4, Med-PaLM 2) on structured, narrow tasks in benchmark settings (88–98% vs. 78–91% accuracy). These figures derive from curated datasets and should not be extrapolated to routine clinical environments. Across 34 studies reporting both benchmark and deployment data, real-world performance declined consistently (5–28% reduction). Hallucination rates were 5–12% for domain-adapted and 15–30% for general-purpose models in generative tasks. Key barriers included data privacy concerns (89%), absent regulatory frameworks (77%), and limited interpretability (83%). LLMs show promise in controlled settings, but evidence is dominated by retrospective evaluations on curated datasets and real-world performance is consistently lower. Responsible clinical integration requires addressing reliability, interpretability, privacy, regulatory readiness, and demographic equity.