Prototype-Based Asymmetric Hierarchical Matching for Text-Video Retrieval

In recent years, CLIP-based text-video retrieval has made significant progress. Most existing text-video retrieval methods usually directly align global video-level and sentence-level features without establishing intrinsic connections between different modalities. Furthermore, due to the asymmetry between visual and textual information, the symmetrical modeling also has inherent limitations that may affect the search performance. To mitigate these problems, we propose an asymmetric hierarchical text-video retrieval framework based on clustering prototypes. Specifically, beyond the commonly used global features, we generate video instance prototypes using clustering algorithms and derive video object prototypes from video patch tokens. The text instance prototypes can be obtained by the developed Text Prototype Generation module (TPG). Besides, the Joint-Interaction Module (JIM) and the coarse-grained instance alignment scheme are proposed to coordinate different modalities. Furthermore, the cross-correlation matrix among instances is proposed, which can further reduce the heterogeneity between video and text modalities. Extensive experiments on several datasets (including MSRVTT, and MSVD) demonstrate that our method outperforms some existing state-of-the-art methods. The source code is available at:https://github.com/rzheng77/PAHM-text-video-retrieval.