A multimodal AI model combining imaging, clinical and biological data significantly outperformed CXR-based AI diagnosis in pneumonia

Background Current AI models that rely solely on chest radiographs (CXRs) or clinical and biological data have limitations. Our goal was to determine if incorporating clinical and biological data to CXR data improved accuracy in the diagnosis of pneumonia. Methods This retrospective study compared three AI models: an imaging model based on a convolutional neural network (CNN) trained on CXRs alone; a clinico-biological model based on a support vector machine (SVM) using clinical and biological data with no CXR; and a multimodal model integrating all three types of information. Data were extracted from two independent cohorts: a training set (PneumOld-CT, n=200, median age 84 years (78.6–90.2)) and an independent test set (PACSCAN, n=230, mean age 65 years +/- 20) for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. We assessed diagnostic performance by the area under the receiver operating characteristic curve (ROC-AUC) and we compared the models using DeLong's test. Calibration curves and decision curve analysis (DCA) were also evaluated. Results In the independent test set, the multimodal AI model demonstrated significantly higher ROC-AUC than the imaging-based model (p < 0.05) or the clinico-biological model (p < 0.005). DCA confirmed a greater net clinical benefit for the multimodal model. Conclusion Integrating radiographic, clinical and biological data to develop a multimodal AI model significantly improved pneumonia diagnosis compared to a single-or a dual modality AI model. This multimodal approach has the potential to improve diagnostic support, especially in complex clinical scenarios.