A head-to-head comparison of GPT-based prognostic predictions and oncologists in gastrointestinal cancers.

808 Background: Patients and families want to know their likely survival times but oncologists have trouble making estimates and communicating this information. Large language models (LLMs) such as ChatGPT may assist with estimating prognosis. Methods: We conducted a retrospective pilot study using clinical data from 22 patients with advanced gastrointestinal malignancies and known survival time. A single progress note from near the time of diagnosis was de-identified and given to a gastrointestinal oncologist and a HIPAA-compliant instance of ChatGPT. Likelihood of survival at 6 months, 1 year, 2 years, and 5 years were estimated and categorized as likely (>75%), possible (25-75%), and unlikely (75% bins consistently corresponded to high observed survival (87–100%), reflecting a tendency toward more optimistic survival estimates, whereas oncologists rarely used high-survival bins and often underestimated survival. Conclusions: Oncologists were numerically more accurate overall, with a signal toward greater accuracy at 1 year, though differences were not statistically significant. ChatGPT achieved comparable performance and demonstrated superior calibration of probability estimates, supporting its potential as a complementary prognostic tool in gastrointestinal oncology.