A Multimodal Feature Fusion Model for Predicting Secondary Loss of Response After Infliximab Treatment in Crohn's Disease Patients: A Multicenter Study

BACKGROUND: The early prediction of secondary loss of response (SLOR) after infliximab (IFX) treatment in Crohn's disease (CD) patients can help optimize treatment strategies. This study developed and validated a multimodal deep learning model that uses baseline endoscopic ulcer lesions to predict SLOR. Additionally, a deep learning-based ulcer detection model was established to automatically identify ulcer lesions. METHODS: A total of 385 CD patients from three centers were retrospectively analyzed. An ulcer detection model was developed to identify endoscopic ulcer lesions from 12,092 endoscopic images. Following lesion localization, 2189 ulcer images were selected and used to train feature fusion models, while clinical data were incorporated to construct a multimodal model for SLOR prediction. These models were validated in two external test cohorts. RESULTS: The ulcer detection model achieved precision values of 0.853 in the validation cohort. The multimodal model outperformed the clinical model in predicting SLOR with areas under the ROC curve (AUCs) of 0.892, 0.847, and 0.824 in the internal validation cohort, external test cohort 1, and external test cohort 2, respectively. Gradient-weighted class activation mapping (Grad-CAMs) revealed highly pronounced activation of the ulcerated area in SLOR patients in the model, providing crucial support for model prediction. CONCLUSIONS: The ulcer detection model effectively identifies ulcer lesions, increasing diagnostic efficiency. The multimodal model, which integrates baseline endoscopic images and clinical data, offers a potential tool for early SLOR prediction.