Spatial–Spectral Operator Mixture Transformer With Mixture-of-Experts Fusion for Endoscopic Classification

Endoscopic image classification is a critical component of computer-aided diagnosis but remains hindered by significant challenges, including inter-patient domain shifts, data scarcity, and a conventional reliance on spatial features that often fail to capture subtle, high-frequency textural anomalies inherent to early-stage pathologies. To mitigate these limitations, we propose SSOMT (Spatial-Spectral Operator Mixture Transformer), a novel dual-stream architecture that explicitly integrates learnable multi-scale frequency-domain representations with hierarchical spatial features through a principled, contentadaptive fusion mechanism. The spatial stream leverages a pretrained DaViT backbone to extract rich local semantic information, while a parallel spectral encoder utilizes learnable Fourier neural operators to perform decomposition at multiple patch scales, isolating amplitude and phase cues that are robust to illumination variations. To effectively synthesize these complementary modalities, we introduce a heterogeneous Mixture-of-Experts (MoE) module that dynamically routes spatial and spectral tokens through learned gating networks. This allows the model to adaptively weigh the importance of texture (spectral) versus shape (spatial) for each specific input, reinforced by MixStyle regularization to improve generalization across diverse imaging domains. Extensive evaluation on two benchmark endoscopic datasets demonstrates the strong performance of the proposed approach. SSOMT achieves 99.43% ± 0.57% accuracy on bladder tissue classification, corresponding to a +0.40 percentage-point gain over the strongest listed baseline, and 94.08% ± 0.57% accuracy on gastrointestinal classification, showing competitive performance under the adopted five-fold protocol. Crucially, the model achieves 100% recall for benign tissue and 99.79% recall for high-grade cancer, reducing missed diagnoses. These results show that integrating spectral operators with spatial transformers yields a robust, clinically reliable framework for precise endoscopic tissue characterization, with stable performance across two benchmark endoscopic datasets overall.