SCL: Semantic Coherence Learning for Video Question Answering

Video Question Answering (VideoQA) requires clarifying the true question intent and understanding dynamic video content to predict the answer. However, most existing methods overlook the temporal semantic exploration of fine-grained clues across frames, and neglect the interrogative properties of question text, leading to suboptimal grounding of video frames. To address these limitations, in this paper, we propose a novel Semantic Coherence Learning (SCL) framework, which explicitly models the fine-grained spatio-temporal coherence of critical visual regions and progressively mines multi-modal semantic clues aligned with the reasoning intent. Specifically, we design a spatio-temporal clue reasoning module to adaptively model the fine-grained coherent transition semantics of critical regions across temporal frames while suppressing irrelevant noisy regions within frames. Additionally, we propose a multi-modal clue reasoning module to progressively mine multi-modal coherent semantics associated with reasoning intent, which iteratively grounds critical frames in videos based on refined question semantics and mines critical phrases in questions based on refined video semantics. Finally, the answer is derived through an answer decoder with intra and inter-sample contrastive strategies. Extensive experimental results on six widely used benchmarks verify our effectiveness and superiority. The experimental source codes are available at https://github.com/XizeWu/SCL.