Rethinking Domain Adaptive Semantic Segmentation With Vision Foundation Models

Unsupervised Domain Adaptive Semantic Segmentation (UDASS) studies the challenge of transferring knowledge from labeled source domains to unlabeled target domains. Existing UDASS methods rely on ImageNet-pretrained Transformer networks, but both the limited scale of ImageNet and the restricted capacity of the pretrained models constrain generalization, ultimately hindering performance. Recently, Vision Foundation Models (VFMs) pre-trained on large-scale datasets have achieved a significant generalization ability on downstream tasks, yet their potential for UDASS remains largely unexplored. Thus, in this work, we rethink and explore the potential of VFMs in UDASS and propose a framework via VFMs for UDASS. Specifically, we first construct a general VFM-based UDASS framework by combining different VFMs with various decoders, and then evaluate its effectiveness on a standard UDASS benchmark under both full-parameter and frozen-parameter fine-tuning paradigms. Then, we propose a generalized parameter-efficient fine-tuning approach containing a Dual Distribution Alignment (DDA) strategy and a Selective Feature Refinement (SFR) strategy. The DDA strategy aligns covariance differences while enforcing semantic consistency, thereby improving source–target distribution alignment. The SFR strategy employs learnable tokens to refine frozen VFM features based on the source–pseudo-target covariance, preserving domain-invariant information while adapting to target-specific cues. Extensive experiments demonstrate the effectiveness of the proposed approach in multiple UDASS benchmarks. The code is released onhttps://github.com/seabearlmx/VFM4UDASS.