Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images

Globally, breast cancer remains a significant health challenge that has a direct effect on women's cancer morbidity and mortality. The estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) are important factors that help doctor to determine the best treatment for each woman. When using immunohistochemistry and genomic assays to look for markers, it is a relatively long and slow process that varies from individual to individual. The aim of this study is to develop a deep-learning framework to predict directly the ER, PR and HER2 status of H&E-stained histopathology images. The technique entails downsampling Level-1 slide images from the TCGA-BRCA cohort, followed by using a pre-trained ResNet50 architecture to extract histological features to enhance the accuracy of biomarker prediction. We train a multi-output classification model using XGBoost that adds a classifier chain. We use a mixture of clinical and genetic data as well as image features. This joint computational method shows promise in enhancing the accuracy of biomarker predictions and enabling doctors to customize breast cancer treatment for individual patients.