Quantile-Masked Softmax Activation for Inception CNN-Based Histopathological Breast Cancer Subtype Classification

Accurate classification of breast cancer subtypes through histopathological image analysis is crucial for clinical diagnosis and treatment planning. However, Convolutional Neural Network (CNNs) models employing standard softmax activation often distribute prediction probabilities across irrelevant logits, potentially diluting discriminative capabilities. To address this limitation, we propose a novel Quantile-Masked Softmax (QM-Softmax) activation function, which selectively emphasizes informative logits through a quantile-based masking strategy. This study systematically evaluates QM-Softmax within an InceptionV1 CNNs framework, using histopathological breast cancer images at 40× magnification across eight distinct subtypes. Experimental results demonstrated significant accuracy improvements (up to 83.21%) with QM-Softmax, especially at carefully tuned thresholds (e.g., 0.001). ROC curve analyses further supported improved subtype-specific classification. These results underline the clinical potential of QM-Softmax activation, emphasizing its effectiveness and necessity for careful parameter selection to optimize breast cancer histopathological image classification.