Grad-CAM-Enhanced Dual-Image Prompting of Multimodal LLMs for Trustworthy Brain Tumor Classification

SUMMARY & CONCLUSIONSMagnetic-resonance imaging (MRI) is indispensable for brain tumor management. However, radiologists must routinely interpret thousands of slices per shift, amplifying fatigue and diagnostic error. While convolutional networks exceed 95% accuracy on public brain tumor benchmarks, their black-box nature, limited calibration, and high computational costs hinder the deployment in safety-critical workflows. Multimodal large language models (LLMs) add free-text rationale but introduce hallucination risk. To bridge this gap, we propose a pipeline that fuses image-level explainability with prompt-engineered multi-modal inference and confidence-based risk control. First, 7,023 T1-weighted slices from the Kaggle Brain-Tumor MRI dataset were pre-processed. Second, an EfficientNet-B0 classifier was trained on the same data; then class-specific Gradient-weighted Class Activation Mapping (Grad-CAM) were utilized for every test slice. Third, GPT-4 Omni mini-high and GPT-o3 models were prompted either with the original slice alone or with a dual-panel montage that paired the slice and its heat-map, requesting tumor class, a maximum-50-word neuroradiology rationale, and a PubMed citation. Finally, the predictions were validated against ground truth, and a calibration-aware gate deferred low-confidence or schema-violating outputs to human review. The dual-image prompt raised macro accuracy/F1 for GPT-4 Omni from 0.64/0.64 to 0.67/0.67, while reducing glioma–meningioma cross-confusion by 12%. A confidence threshold set to keep the false-negative rate below 2% allowed 68% of cases to be automated with zero residual errors. Results generalized to GPT-o3, confirming model-agnostic benefit. The study demonstrates that prompt-level augmentation—without LLM retraining—can enhance accuracy, interpretability, and maintainability, offering a practical blueprint for trustworthy artificial intelligence (AI) decision support in neuro-oncology and other regulated domains.