AI vs the heart: can chatGPT-4o reliably detect cardiomegaly on chest X-Rays?

Abstract Background Large language models (LLMs) have shown promise in clinical decision support, yet their image interpretation capabilities remain under-explored. Emerging trends of use in clinical environments without validation is of concern, given the potential to harm patients by providing inaccurate information to overly trusting clinicians [1]. This pilot study assessed ChatGPT-4o’s ability to assess the presence or absence of cardiomegaly in an established chest x-ray (CXR) database to assess its current potential for clinical use. Method ChatGPT-4o was prompted to provide binary, "yes/no" answers for a diagnosis of cardiomegaly from cases drawn from a National Institute of Health’s (NIH) open-access chest x-ray database, with 50 confirmed cases, 25 confirmed normal cases, and 25 cases with diagnoses other than cardiomegaly [2]. Statistical analysis of inter-rater agreement, as well as McNemar’s test of paired proportions, significant difference, and bias assessment (i.e., propensity to over-/under-call a diagnosis) was completed using an institutional access version of StataMP (StataCorp. 2025. Stata Statistical Software: Release 19. College Station, TX: StataCorp LLC). Results Inter-rater agreement (Cohen’s Kappa) for cases with a confirmed diagnosis of cardiomegaly (n = 50) was 0.48. McNemar’s tests of significant differences (Chi-squared) between the two pairs across asymptotic, marginal homogeneity (Stuart-Maxwell, Bickenboller, no diagonals), and linear trend assessment was 26.0 (p =