Multimodal Deep Learning for Breast Cancer Subtype Classification Using Histopathology and Biomarker Data

Accurate diagnosis and subtype classification are essential for directing treatment strategies for breast cancer. The gold standard for breast cancer classification is the analysis of histopathological images. Biomarkers offer insights for prediction since they are indicators of the state of the disease. Deep learning has achieved significant progress in breast cancer subtype classification, with studies exploring both histopathology-based and multimodal approaches. But, to our knowledge, none of the studies explore the advantage of integrating biomarker data to boost the model’s reliability by being able to identify patterns in the true complexity of the biology. To address this gap, we develop and compare unimodal (histopathology-only) and multimodal (histopathology + biomarker) deep learning models. To do this, we selected 600 breast cancer samples for this study, representing four main subtypes: Triple-Negative Breast Cancer (TNBC), Invasive Lobular Carcinoma (ILC), Invasive Ductal Carcinoma (IDC), and Metaplastic Breast Cancer (MBC). Based on their diagnostic and prognostic value, totally thirteen clinically relevant biomarkers, i.e., Ki-67, EGFR, ESR1, PGR, BRCA1, TP53, ERBB2 (HER2), DH1, TP53, ERBB2, RB1, SNAI1, PTEN, were chosen for the subtypes. After the training and testing of each model, the multimodal framework reflected an approximately 87% relative improvement of the macro-F1 over the unimodal image-only model, demonstrating the clear benefit of the addition of biomarker data into the classification process.