DHSA: dual hierarchical semantic alignment for visual question answering

Visual Question Answering (VQA) requires a deep understanding of multimodal semantics and the ability to effectively integrate task-relevant information from both visual and textual modalities. While existing approaches predominantly rely on attention mechanisms for answer reasoning, they often fail to fully leverage semantic-level representations of different modalities. To address this, we propose the Dual Hierarchical Semantic Alignment (DHSA) model, which tackles two fundamental challenges: 1) insufficient semantic alignment due to heterogeneous modality gaps, and 2) interference from conflicting information during multimodal fusion. The DHSA model constructs both private and shared semantic spaces through cross-modal semantic projection, enabling a deeper understanding of multimodal semantics. It further employs contrastive learning to align semantically consistent image-question pairs while distancing inconsistent ones. During the fusion stage, self-attention and guided attention mechanisms are utilized to achieve effective multimodal interaction. Experiments on the VQAv2 dataset indicate that our method performs favorably compared to several strong baselines, with noticeable improvements in accuracy observed across multiple settings.