Symmetry-aware Mod-Seg-SE(2) framework for MRI brain tumor segmentation

Brain tumor segmentation in Magnetic Resonance Imaging (MRI) remains a critical task in neuro-oncology, requiring precise delineation of heterogeneous tumor shapes and anatomical variability. We introduce Modified Segmentation Special Euclidean 2 (Mod-Seg-SE(2)), a novel roto-translationally equivariant geometric deep learning architecture designed to enhance spatial consistency and boundary preservation in tumor segmentation. Built upon geometric deep learning principles, the model incorporates lifting layers, SE(2) group convolutions, and Squeeze-and-Excitation (SE) recalibration within a U-Net-inspired encoder-decoder design. This architecture embeds roto-translation symmetry directly into the network, enabling robust spatial feature extraction and transformation consistency. The decoder employs group deconvolutions to reconstruct segmentation masks equivariantly across both spatial and orientation dimensions. We evaluate the model on private clinical brain tumor datasets, achieving a Dice score of 0.635 and IoU of 0.715, significantly outperforming U-Net and No New U-Net (nn U-Net) baselines. To assess generalizability beyond neuroimaging, we further tested Mod-Seg-SE(2) on a publicly available blood cell segmentation dataset. The model achieved a Dice score of 0.8788 and IoU of 0.8223, surpassing U-Net (Dice: 0.6983, IoU: 0.6831) and nn U-Net (Dice: 0.7518, IoU: 0.7307). These results demonstrate that Mod-Seg-SE(2) not only improves boundary accuracy in brain tumor segmentation but also generalizes effectively across domains with different structural and orientation properties. Moreover, the model reduces reliance on data augmentation and lowers inference time, supporting real-time deployment in clinical workflows. These findings validate Mod-Seg-SE(2) as a robust, accurate, and interpretable solution for medical image segmentation tasks in both specialized and general biomedical imaging pipelines.