Dual-Branch Collaborative Implicit-Explicit Mutual Learning for Weakly Supervised Video-Sentence Retrieval and Grounding

Video-Sentence Retrieval and Grounding (VSRG) task aims to retrieve the corresponding video from a video corpus based on a single sentence query and accurately localize the temporal boundary of the sentence within the video. However, employing an end-to-end joint optimization strategy faces a critical challenge in the VSRG task: a task conflict exists between the retrieval and grounding tasks. Specifically, the retrieval task focuses on global video-text matching, while the grounding task requires fine-grained alignment between a local video segment and the sentence query. To address this issue, we propose a Dual-Branch Collaborative Implicit-Explicit Mutual Learning (DCIML) framework. The framework adopts a dual-branch structure, where the retrieval branch is responsible for cross-modal video-text retrieval, and the grounding branch achieves precise temporal grounding based on the sentence query. Furthermore, we employ an iterative optimization strategy to train each task. Meanwhile, to promote collaborative optimization of the dual tasks, we design a dual-branch mutual learning module that facilitates cross-task knowledge transfer through bidirectional explicit and implicit pathways. The DCIML framework not only effectively resolves the task conflict between the retrieval and grounding tasks but also achieves collaborative optimization of the dual tasks. Experimental results demonstrate that DCIML performs excellently on the ActivityNet-Captions and Charades-STA datasets, validating its effectiveness. The code is available athttps://github.com/X7J92/DCIML.