S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models

Recent works enhance the zero-shot generalization of vision-language models (e.g., CLIP) by alleviating semantic misalignment between image and text embeddings. However, they largely overlook lexical variation, where a class can be described by diverse textual concepts, leading to unstable alignment. We propose a Synonymous Semantic Space (S3) that constructs a compact continuous space of synonymous concepts generated by large language models. Furthermore, S3enables robust point-to-space similarity with a novel point-to-local-center metric, yielding effective zero-shot predictions. Extensive experiments on 15 benchmarks, including fine-grained and natural distribution zero-shot classification, demonstrate that S3consistently outperforms state-of-the-art methods.