PolySAM-Lite: Parameter-efficient adaptation of the Segment Anything Model for colorectal polyp segmentation

Abstract Colorectal cancer is a leading cause of global cancer mortality, where early detection and segmentation of polyps during colonoscopy are critical for survival. While Vision Foundation Models like the Segment Anything Model (SAM) demonstrate exceptional zero-shot generalization, their massive computational footprint hinders deployment in resource-constrained clinical settings. To bridge this gap, we introduce PolySAM-Lite, a resource-efficient framework that adapts the SAM architecture for specific medical segmentation tasks using Low-Rank Adaptation (LoRA). By freezing the heavy image encoder and injecting trainable low-rank matrices specifically into the fused Query-Key-Value (QKV) attention layers, we fine-tuned the model using only 4.2 million parameters (∼4.5% of the ViT-Base total). Experimental evaluations on the Kvasir-SEG dataset demonstrate that PolySAM-Lite achieves a Dice Similarity Coefficient (DSC) of 0.9348, significantly outperforming the zero-shot SAM baseline (DSC: 0.8656) by 6.92%. Furthermore, ablation studies reveal that our method maintains robust performance (DSC: 0.9240) even when trained on only 50% of the available data. Statistical analysis confirms the significance of these improvements (p < 0.001). Notably, PolySAM-Lite achieved an Area Under the Curve (AUC) of 0.9972 on a single consumer-grade NVIDIA T4 GPU, demonstrating that high-performance medical AI can be democratized for low-resource healthcare environments.