LMMs for histopathology: zero- and few-shot patch classification with GPT and Gemini models

Large multimodal models such as GPT and Gemini have shown promise in general vision language tasks, yet their direct application to histopathology, particularly for Whole Slide Image patch classification, remains largely unexplored. We present the first systematic comparison of GPT and Gemini models across three diagnostically distinct domains: Ziehl-Neelsen stained tuberculosis detection (AFB), Pap smear based cervical cytology (CRIC), and H&E-stained breast tumor subtyping (BreaKHis). We evaluate zero shot binary classification, binary versus multiclass prompting, minimally fine-tuned settings, and majority voting strategies. GPT models consistently outperformed Gemini counterparts, with GPT-4.1 achieving the highest average accuracy of 0.67. In zero shot evaluation, GPT-4.1 reached 0.85 on AFB and 0.62 on CRIC, while Gemini 2 Flash achieved 0.57 on BreaKHis. Multiclass prompting improved CRIC binary accuracy from 0.575 to 0.644. Fine-tuned AFB GPT-4o trained on 100 samples achieved 0.93 on its native task. Fine tuning on the high contrast AFB task produced robust cross domain transfer, outperforming the BreaKHis tuned model on BreaKHis with scores of 0.59 and 0.57 compared to 0.52.