MGMMD: Multi-Grained Multi-Modal Diffusion Model for Text-Video Cross-Modal Retrieval

Most existing text-video retrieval works map text and video into the same latent semantic space for direct similarity calculation, without exploring the multi-modal information of the video or the relevant text information of videos that can be used to match query text, such as descriptions of video content. To tackle this problem, we propose a novel multi-grained multi-modal diffusion model for text-video cross-modal retrieval (MGMMD). Firstly, our proposed model performs multi-grained text-video matching which fuses the global coarse-grained multi-modal matching, fine-grained multi-modal matching based on multi-modal diffusion model and supplementary additional text matching, aiming at conducting more comprehensive text-video matching. Secondly, a multi-modal diffusion model with cross-attention mechanism is proposed to reason on fine-grained multi-modal text-video matching to sufficiently learn the rich multi-modal features such as image, text, and audio features in videos for better cross-modal retrieval. Fine-grained multi-modal matching module based on multi-modal diffusion model models the cross-modal retrieval task as a process of gradually generating joint distributions from noise through a multi-modal diffusion-based framework. Thirdly, we propose to generate supplementary additional captions which can also represent the video content to further enhance existing text-video retrieval. Extensive experiments conducted on three widely used benchmark datasets demonstrate the effectiveness of the proposed model by comparing it with state-of-the-art methods.