CompoVis: Is Cross-Modal Semantic Alignment of CLIP Optimal? A Visual Analysis Attempt

Vision-language pre-trained models (VLMs) have shown impressive cross-modal understanding, yet their “compositional understanding” ability remains under investigation. We introduce CompoVis, a framework for visually probing cross-modal gaps in VLMs. CompoVis optimizes the grid layout to highlight alignment clusters and boundaries, visually interprets multi-head attention and semantic drift, and enables interactive fine-tuning unconstrained by closed datasets or offline models. Quantitative experiments and case studies explore key insights: VLMs rely on entity shortcuts rather than comprehension-driven; stubborn global modality isolation and suboptimal fine-grained alignment remain; fine-tuning with negative samples does not fundamentally alleviate the gaps. Approximately 89% of participants ($n=27$) found that, compared to methods relying solely on data metrics, CompoVis offers a more innovative and effective approach for investigating modality gaps in VLMs.