Benchmarking LLMs for hospital-course summarisation: aligning metrics with clinical factuality, safety, and robustness

To the Editor, We read with interest Aali et al.’s study introducing MIMIC-IV-BHC and a benchmark of large language models (LLMs) for brief hospital-course (BHC) summaries.1 The work makes a timely contribution by releasing resources and reporting baselines across context lengths and adaptation strategies. We suggest several additions that could further strengthen the benchmark’s clinical validity, reproducibility, and generalizability. In sum, Aali et al. provide an important foundation for BHC summarization benchmarking. Augmenting evaluation with factuality/safety metrics, more informative reader-study reporting, robustness stress tests, and external/multilingual validation would further align the benchmark with clinical veracity and translational readiness. Jose Zablah (Methodology, Writing–original draft), Yolly Molina (Validation, Writing–original draft), and Antonio Garcia Loureiro (Conceptualization, Writing–review & editing) None declared. None declared. No new datasets or code was generated for this article. Data discussed here are publicly hosted as follows: the MIMIC-IV-Ext-BHC labeled notes resource for hospital-course summarization (Version 1.2.0; https://doi.org/10.13026/5gte-bv70) is available to credentialed users on PhysioNet; it derives from MIMIC-IV-Note (Version 2.2; https://doi.org/10.13026/1n74-ne17) and is linkable to the MIMIC-IV clinical database (Version 3.1; https://doi.org/10.13026/kpb9-mt58). Access policies (DUA and CITI training) are detailed on the PhysioNet project pages. The JAMIA article introducing the benchmark is available at https://doi.org/10.1093/jamia/ocae312, and the accompanying benchmarking scripts are maintained at https://github.com/StanfordMIMI/clin-bhc-summ. Evaluation tools referenced in the text are openly available: QAFactEval (https://github.com/salesforce/QAFactEval) and finally CheckList https://github.com/marcotcr/checklist).