Transferable Backdoor Attack on Any CLIP Model With Any Target Class by Pre-Trained Hack Network

CLIP has promoted miscellaneous models and downstream applications, creating a great family in the deep learning community. Recently, the security of CLIP has become an increasingly popular research topic, where backdoor attacks show potential devastation by injecting a covert backdoor to manipulate the output features. Existing backdoor attacks typically bind the target class with large-scale poisonous data to pre-train or finetune the target CLIP model. Such a paradigm lacks transferability and flexibility since it needs retraining when applied to another target class or target CLIP model. In this paper, we propose a novel Transferable Backdoor Attack (TBA) paradigm for transferring backdoor attacks on any CLIP model with any target class, reducing time costs from conventional days or hours to seconds. First, we propose a multimodal hack network pre-training step, where the visual and textual hack networks are pre-trained to recognize the triggers in multimodal inputs robustly. Second, we propose a few-shot training-free backdoor transfer step, where the hack networks are efficiently fused into the target CLIP model with few samples of the target class. In experiments, our Transferable Hack Network (THNet) consistently achieves$>99$% attack success rate while making minimum impact for clean input on various TBA settings and two downstream tasks. We hope this work can contribute a new perspective on the security vulnerability of CLIP and its downstream applications.