Multi-Modal Knowledge Distillation Hashing Based on CLIP for Weakly Supervised Image Retrieval

Existing weakly supervised hashing often suffers from the imprecision of user-provided tags and over-reliance on textual knowledge from pre-trained word embeddings, neglecting crucial visual knowledge associated with image labels. As a result, this leads to unsatisfactory performance in closed-vocabulary tasks and limited generalization in open-vocabulary scenarios. To address this issue, we propose Multi-modal Knowledge Distillation Hashing (MKDH), a novel method leveraging visual and language pre-training (VLP) model such as CLIP to learn robust hash codes. Our method designs a dual-layer attention adapter to generate joint representations by capturing fine-grained visual and textual knowledge from the CLIP teacher network. Additionally, we introduce a knowledge extraction contrastive loss to enhance the robustness of joint representations and a knowledge distillation contrastive loss to transfer the extracted multi-modal knowledge to the hash codes. To further mitigate the negative impact of false negative pairs in these contrastive losses, we introduce false negative weighting strategy that reduces the weights assigned to such pairs. Extensive experiments on three widely used datasets demonstrate that our method achieves robust retrieval performance with significant improvements in both closed- and open-vocabulary settings. The source code is available athttps://github.com/IMAG-LZY/MKDH.