Question Understanding and Temporality Guiding for Video Question Answering

Video Question Answering (VideoQA) aims to answer a question based on the content of a given video. Recent methods adapt image-text pre-trained models to the VideoQA task by designing learnable temporal modules within the image encoder. However, these methods struggle to fully comprehend the questions and effectively extract temporal information due to 1) over-reliance on candidate answers and 2) lack of explicit temporal modeling. Specifically, since the question is fixed in different question-answer pairs, existing models tend to focus on the varying candidate answers. Moreover, existing methods merely utilize the classification loss to constrain the confidence of candidate answers, failing to differentiate the effectiveness of temporal information and to explicitly guide temporal modeling. In this paper, we introduce the Question Understanding and Temporality Guiding (QU-TG) method to address the aforementioned limitations. To reduce over-reliance on candidate answers, we propose providing diverse questions through question selection and enhancing the model's comprehensive understanding of questions through question-video matching. To conduct explicit temporal modeling guiding, we propose negative video prevention and positive video guidance to conduct explicit temporal modeling guiding. Negative video prevention incorporates a prevention loss to discourage the model from making predictions based on erroneous temporal cues, whereas positive video guidance utilizes classification loss to encourage the model to derive correct answers from positive videos. Extensive experiments on the NExT-QA, IntentQA, STAR-QA, and Causal-VidQA datasets demonstrate the effectiveness and generalization of our method.