Beyond Mere Tuning: Harnessing the Full Potential of Prompts for Text-Video Retrieval

Text-video retrieval has advanced significantly by thanks to the merits of large-scale contrastive languageimage pre-training. Recently, prompt tuning has emerged as a parameter-efficient strategy to solve the substantial training costs associated with these models. However, existing studies are generally confined to simple tuning strategies, without fully exploring the potential of prompts. Unlike its application to singlemodal tasks, prompt tuning in cross-modal learning involves not only the adaptation to specific tasks but also the challenge of dealing with the intricate interaction of semantic information. These aspects remain insufficiently explored, thereby limiting retrieval performance. To this end, we introduce a comprehensive framework for harnessing the full potential of prompts in textvideo retrieval. Specifically, we explore prompts in three respects: i) Task adaptation: we design appropriate prompts to guide the fine-tuning of pre-trained models to perform text-video retrieval tasks. ii) Temporal modelling: to enable temporal modelling, while protecting the original knowledge from being corrupted, we propose a prompt rolling module that enables effective temporal processing without adding extra training parameters. iii) The sampling scope prediction: We introduce a query expansion strategy that exploits the rich semantic information conveyed by multimodal prompt features to define the sampling scope more precisely. Extensive experiments on four mainstream datasets, i.e., MSR-VTT, ActivityNet, DiDeMo, and LSMDC, demonstrate that our method achieves state-of-the-art performance.