The Diminishing Returns of Scale: A Systematic Evaluation of Segment Anything Model Variants Across Image Characteristics

The Segment Anything Model (SAM) represents a paradigm shift in image segmentation, offering three model variants of increasing size: ViT-B (91M parameters), ViT-L (308M), and ViT-H (636M). While larger models are presumed to deliver superior performance, the practical trade-offs between accuracy gains and computational costs remain poorly characterized. This study presents a systematic evaluation of all three SAM variants across 200 medical images from the Kvasir-SEG dataset, measuring Intersection over Union (IoU), Dice coefficient, boundary F1 score, and inference time. Our results reveal that ViT-H achieves only a 1.27% improvement in mean IoU over ViT-B (0.794 vs. 0.784), a difference that is not statistically significant (p > 0.05, paired t-test). However, ViT-H requires 4.0× longer inference time (460ms vs. 115ms per image), resulting in ViT-B delivering 3.9× better efficiency (IoU per second). Stratified analysis reveals that larger models provide marginal benefits primarily for very large objects (>30% of image area) and demonstrate superior robustness to image degradation, particularly Gaussian noise (2.7% IoU drop for ViT-H vs. 4.4% for ViT-B). These findings suggest that for most practical segmentation tasks, the base ViT-B model offers optimal performance-efficiency trade-offs, with larger variants justified only in specific scenarios requiring maximum robustness or handling of very large objects.