Adaptability of Vision Foundation Models for 3D Medical Image Segmentation

Vision Foundation Models (VFMs), such as DINOv2 and SAM, have demonstrated unprecedented generalizability in natural imaging and show strong promise in medical imaging due to their semantically rich representations. However, their effective application to 3D volumetric segmentation remains largely underexplored, especially concerning optimal adaptation strategies for transferring 2D pretrained knowledge to the structurally disparate 3D domain. To address this, we present a comprehensive investigation into the transferability and task-specific adaptability of six diverse 2D VFMs (including Self- Supervised, Vision-Language, and Segmentation Generalists) for 3D medical image segmentation. We systematically evaluate four distinct transfer learning paradigms, including advanced Fine-Tuning methods, across four heterogeneous 3D medical datasets. Our results establish VFMs as a powerful and cost-effective generalist baseline, consistently outperforming non-pretrained and standard 3D ViT architectures despite the substantial domain shift. Crucially, our systematic exploration reveals that parameter-efficient finetuning achieves the highest segmentation accuracy across all datasets. Feature-level analyses using PCA and CKA provide key insights, confirming that optimal performance stems from successfully balancing the preservation of generalizable low-level visual features with the adaptation of high-level, task-specific semantics.