Zero-Shot Tokenizer Transfer for Targeted Attacks on Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is widely used to improve the reliability of Large Language Models (LLMs) by grounding responses in external documents. However, RAG systems that ingest content from internet sources remain vulnerable to prompt injection attacks embedded within retrieved documents. We introduce ZBEAST, an adaptation of Beam Search-Based Adversarial Attacks (BEAST), designed to generate malicious documents specifically for RAG settings.We evaluate ZBEAST using an embedding-based heuristic built on BAAI-BGE-base-v1.5 and compare it against the state-of-the-art BEAST approach, which relies on an instruction-tuned LLM during search. ZBEAST replaces the generator’s tokenizer with that of the embedding model via Zero-Shot Tokenizer Transfer (ZeTT), enabling more effective adversarial document optimization. Our results show that ZBEAST scales reliably with beam size, while the baseline approach exhibits diminishing returns. We further demonstrate that ZBEAST-generated documents evade common filtering mechanisms used in RAG pipelines. These findings expose an underexplored vulnerability in RAG systems and suggest existing defenses may be insufficient against optimized adversarial document generation.