OERViT: Optimized Ensemble ResNet50 Vision Transformer for Prostate Cancer Diagnosis

Diagnosing prostate cancer from multi-parametric MRI (mpMRI) remains a clinical challenge due to limited annotated datasets, significant inter-reader variability, and the necessity to capture both local lesion details and broader anatomical context. To address these challenges, we propose OERViT (Optimized Ensemble ResNet50-Vision Transformer), a hybrid deep learning framework that combines CNN-driven local feature extraction with Transformer-based global context modeling specifically designed for multi-sequence mpMRI prostate cancer classification. The architecture employs ResNet-50 to extract 2048-dimensional hierarchical spatial features from five mpMRI sequences (T2-weighted sagittal, coronal, transverse, diffusion-weighted imaging, and apparent diffusion coefficient maps), while a custom Vision Transformer branch with 8-head self-attention across 4 encoder layers captures long-range anatomical relationships through a 512-dimensional global representation. To improve robustness and reduce optimizer-related bias, three model variants were trained using different optimization algorithms (Adam, Lookahead+RAdam, and Lion), with predictions combined through majority voting to leverage complementary convergence behaviors. Evaluated on the ProstateX dataset (346 patients) with stratified data splitting, OERViT achieved 85.71% accuracy, F1-score of 0.7826, and AUC of 0.8406 on the held-out test set, demonstrating 5.71% absolute improvement in accuracy, 12.5% in F1-score, and 13.7% in AUC compared to a baseline ResNet-50 model. Five-fold stratified cross-validation confirmed model stability with a mean AUC of 71.37% (±5.65%) for the best-performing optimizer configuration. These results demonstrate that integrating complementary CNN-Transformer architectures with multi-optimizer ensemble strategies offers a practical and effective approach for mpMRI-based prostate cancer classification under limited data constraints, achieving competitive performance with state-of-the-art methods while maintaining computational efficiency suitable for clinical deployment.