An Electronic Medical Record-Embedded Large Language Model for Acute Pancreatitis Diagnosis, Severity, and Prognosis (Preprint)

UNSTRUCTURED Background: Early diagnosis, accurate severity assessment of acute pancreatitis (AP), and prediction of progression to severe acute pancreatitis (SAP) are critical. We evaluated an electronic medical record (EMR)-embedded large language model (LLM) for these tasks. Methods: The LLM reviewed earliest AP hospitalization records of 261 adults and answered three prompts (diagnosis, severity, and risk of progression to SAP). Results: 224 (85.8%) had mild AP (MAP), 30 (11.5%) moderately SAP (MSAP), and 7 (2.7%) SAP. The LLM diagnosed AP with 89.3% sensitivity and 100.0% positive predictive value (PPV). Severity classification was inconsistent (MAP sensitivity 49.1%, MSAP 66.7%, SAP 42.9%). For progression prediction from initial MAP, the LLM showed high sensitivity (87.5%) but low accuracy (26.8%); Bedside index for severity in acute pancreatitis (BISAP) had higher accuracy (95.5%) but low sensitivity (12.5%). In MSAP, the LLM sensitivity was 85.7% versus BISAP 0%. Conclusions: An EMR-embedded LLM can detect AP and identify many who progress to SAP, but specificity and severity classification require improvement.