UniBreak: A Unified Evolutionary Token-Level Jailbreaking Framework for Large Language Models

Large Language Models (LLMs) demonstrate promising capabilities in natural language understanding and reasoning with enormous parameter spaces and vast amounts of training data. These attributes have facilitated their deployment into diverse application domains. However, the underlying parameters implicitly assume decision-making boundaries, resulting in a significant number of decision spaces not covered by training data. This makes them susceptible to adversarial manipulations through carefully crafted inputs. To illuminate the vulnerabilities of LLMs, we propose a unified token-level jailbreaking attack that makes victim models generate responses for potentially harmful queries. Specifically, we propose an evolutionary algorithm to evolve perturbation sets, utilizing gradient-based and crossover-based operators to enhance performance under multiple scenarios. Furthermore, we develop a repository for reusing past perturbations and conduct an analysis of token sensitivity within LLMs, facilitating zero-shot attacks with convergence. Extensive benchmark experiments validate the effectiveness of our method on three different models, achieving increases of 62.36%, 57.89%, and 64.81% in attack success rates compared to baseline methods under white-box scenarios. In addition, the evaluation experiments demonstrate our method is effective for multiple scenarios and different size of models. This research reveals vulnerabilities of LLMs, provides theoretical foundations for developing more robust defense strategies, and contributes to building more reliable AI systems.