Multi-Granularity Token Learning for Domain Generalized Semantic Segmentation

Recent Domain Generalized Semantic Segmentation (DGSS) methods have harnessed vision foundation models (VFMs) to achieve remarkable performance in unseen domains. Two key issues require immediate attention from these methods. First, these methods ignore the learning of contextual information, since VFMs are always pretrained on the image classification task, which means that the frozen backbone of these VFMs struggles to extract contextual information used for dense prediction tasks. Second, these methods focus on enhancing domain-invariant information within the learnable tokens or the features extracted by the frozen backbone (e.g., normalizing their style variances), inevitably losing the effective information. To address these two key issues, this paper proposes a Multi-Granularity Token Learning (MGTL) approach to fine-tune VFMs for DGSS, which uses the visual and text embeddings from the CLIP model for optimizing a set of learnable tokens to learn the domain-invariant information of two granularities, including the domain-invariant contextual and conceptual information. Specifically, first, a contextual relation learning strategy is proposed to assist tokens in capturing and embedding contextual information into VFMs, which is crucial to the semantic segmentation task. Second, a conceptual learning strategy is proposed to acquire instance-level conceptual information for the set of learnable tokens and embed it into VFMs to enhance instance-level domain-invariant information. The proposed MGTL approach is evaluated on multiple DGSS benchmarks, achieving superior performance to recent DGSS methods while maintaining low computational cost in trainable parameters during the fine-tuning stage.