Stealthy Backdoor Carriers: The Threat of Visual Prompts to CLIP

Visual Prompt (VP) learning has rapidly emerged as a popular paradigm for parameter-efficient task adaptation in CLIP-based models. However, while VP optimizes pixel-space vectors without altering CLIP’s internal weights, this decoupled design inadvertently introduces critical security vulnerabilities. Attackers can exploit VP to implant covert backdoors using imperceptible trigger patterns that bypass traditional anomaly detection mechanisms, posing significant risks to real-world applications. Existing backdoor techniques, however, rely on visually noticeable patterns and exhibit inconsistencies in representation alignment, which make them prone to detection and ineffective against robust defenses. In response to these limitations, we propose Stealthy Backdoor Carriers (SBC), a novel attack framework that leverages CLIP’s inherent vulnerabilities to covertly and persistently inject backdoors.SBCadopts a dual-constrained optimization strategy that balances imperceptibility—minimizing trigger perturbations for visual stealth—and cross-modal embedding alignment—ensuring poisoned and target samples share consistent representations within CLIP’s multimodal space. Experimental results across five benchmark datasets demonstrateSBC’s exceptional effectiveness, achieving a +49.63% improvement in attack success rate relative to existing methods while maintaining robustness against advanced defenses like Neural Cleanse. Our work highlights the need for reevaluating the security implications of VP learning frameworks and provides valuable insights for mitigating prompt-based vulnerabilities in AI systems. Our code is available at https://github.com/Maozhen-Zhang/sbc.git.