Robust Vision-Language Alignment Using Multi-Modal Large Language Models for Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation (OVSS) aims to segment objects without being constrained by a predefined set of categories. Recent advancements in OVSS have been driven by CLIP, a powerful vision-language model that enables segmentation through textual and visual feature matching. However, while visual features within a class exhibit significant diversity, their corresponding text features remain relatively limited in distribution. This discrepancy weakens the effectiveness of vision-language matching forOVSS. To address these challenges, we proposeOV-EVA(Open-Vocabulary Semantic Segmentation with Enhanced Vision-Language Alignment), a novel training-free OVSS framework that leverages multi-modal large language models (MLLMs) to improve visual-language alignment and enhance segmentation robustness. Our method introduces an iterative vocabulary expansion strategy, where MLLMs generate a diverse and scene-relevant vocabulary set through a two-stage querying process. To ensure accurate segmentation, we incorporate a mask-guided score refinement mechanism, which enhances vocabulary terms closely aligned with the target mask while mitigating overly dominant terms across the entire image. Additionally, we introduce a top-N-based target class mapping strategy to improve the alignment between the generated vocabulary set and target class labels. The proposed OV-EVA achieves state-of-the-art performance across multiple benchmarks when using GPT-4o as the underlying MLLM. Furthermore, our approach demonstrates strong adaptability across different MLLMs, achieving competitive results with LLaVA and Janus-Pro. These findings suggest that leveraging generative reasoning from MLLMs offers a scalable pathway for robust zero-shot perception without task-specific training. Future work will focus on optimizing computational efficiency to facilitate real-time deployment in practical applications. We will soon release our source code.