Prompt Injection Attack to Tool Selection in LLM Agents

Tool selection is a key component of LLM agents.A popular approach follows a two-step processretrieval and selection -to pick the most appropriate tool from a tool library for a given task.In this work, we introduce ToolHijacker, a novel prompt injection attack targeting tool selection in no-box scenarios.ToolHijacker injects a malicious tool document into the tool library to manipulate the LLM agent's tool selection process, compelling it to consistently choose the attacker's malicious tool for an attacker-chosen target task.Specifically, we formulate the crafting of such tool documents as an optimization problem and propose a two-phase optimization strategy to solve it.Our extensive experimental evaluation shows that ToolHijacker is highly effective, significantly outperforming existing manualbased and automated prompt injection attacks when applied to tool selection.Moreover, we explore various defenses, including prevention-based defenses (StruQ and SecAlign) and detectionbased defenses (known-answer detection, DataSentinel, perplexity detection, and perplexity windowed detection).Our experimental results indicate that these defenses are insufficient, highlighting the urgent need for developing new defense strategies.