Hybrid Transformer–CNN with Boundary-Aware Attention for Accurate Multi-Modal Brain Tumor Segmentation

Background: Accurate segmentation of brain tumors from multi-modal magnetic resonance imaging (MRI) is essential for diagnosis, treatment planning, and therapy monitoring. However, this task remains challenging due to tumor heterogeneity, irregular boundaries, and the complex anatomical structure of surrounding tissues. In particular, precise delineation of tumor sub-regions—including whole tumor, tumor core, and enhancing tumor—continues to be a major limitation of existing automated methods. Methods: In this study, we propose a novel hybrid CNN–Transformer framework that integrates local feature extraction with global contextual modeling for improved brain tumor segmentation. The architecture consists of three main components: a dual-pathway encoder for capturing fine-grained and contextual features, a multi-scale feature fusion module based on spatial pyramid pooling with dense connections, and a boundary-aware attention decoder designed to enhance segmentation accuracy around tumor edges. The model utilizes four MRI modalities (T1, T1ce, T2, and FLAIR) to capture complementary tumor characteristics. In addition, a hybrid loss function combining Dice, focal Tversky, and boundary losses is employed to address class imbalance and improve boundary precision. Results: Experimental results on the BraTS 2023 dataset demonstrate superior performance, achieving Dice scores of 92.3%, 88.7%, and 84.5% for whole tumor, tumor core, and enhancing tumor, respectively, while maintaining high computational efficiency. Conclusion: The proposed framework achieves accurate and robust brain tumor segmentation by effectively integrating local and global features, demonstrating its potential for automated multi-modal MRI analysis in clinical practice.