Multimodal Brain Tumor Classification Using BLIP and CLIP: A Comparative Study on Accuracy and Performance

Vision-language modelling has significantly shifted image classification paradigms with the emergence of modality pre-training as a very efficient feature-extracting mechanism. In this paper, we intend to fine-tune and compare BLIP and CLIP in a four-grade brain tumor classification task that categorizes MRI scans into glioma, meningioma, and pituitary tumors and no tumor. Embeddings extracted from vision encoders (ViT-based), aligned with fixed class-label prompts (“a photo of a [class] brain MRI”) via cosine similarity. High-dimensional embeddings containing semantics of interest at both local and global levels are extracted by the vision encoders of these models. Various preprocessing techniques, including contrast normalization and standard image augmentations, have been employed to increase domain applicability. Supervised fine-tuning targets vision encoders (last 2 layers unfrozen; text frozen). Fine-tuning was performed using the AdamW optimizer with cosine annealing to keep stable gradient dynamics and achieve optimal convergence. Unlike the previous studies following a zero-shot classification paradigm, this study fine-tunes only CLIP and BLIP for maximizing their applicability in domain-specific studies in medical imaging. Evaluation metrics such as top-k accuracy, precision-recall, and F1-score analyses show that CLIP has the highest accuracy ever (97.94%) on the test set, while BLIP is better contextual representation. These results endorse the utility of vision-language models in the more traditional vision-based medical image analysis sphere and highlight the importance of supervised fine-tuning for better classification performance.