Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation

The clinical utility of deep learning for medical image segmentation is hindered by poor generalization, as models often learn spurious correlations between anatomy and domain-specific styles. To address this, we introduce Causal-SAMLLM, a framework that leverages Large Language Models (LLMs) for causal reasoning atop a frozen Segment Anything Model (SAM) encoder. Our approach introduces two synergistic components. First, Linguistic Adversarial Disentanglement (LAD) uses a Vision-Language Model to generate textual descriptions of confounding styles and contrastively trains the segmentation features to be style-invariant. Second, Test-Time Causal Intervention (TCI) allows an LLM to interpret a clinician’s natural language commands to modulate decoder features for real-time error correction. On a composite benchmark from four public datasets (BTCV, CHAOS, AMOS, BraTS), we evaluate out-of-distribution (OOD) generalization across scanners, modalities, and anatomies. Causal-SAM-LLM sets a new state-of-the-art, outperforming the strongest baseline by up to 5.2 points in Dice score and 10.0 mm in Hausdorff Distance, while fine-tuning under 8% of the total parameters. Our work charts a path toward robust, efficient, and interactively controllable medical AI.