NeuroStrike: Neuron-Level Attacks on Aligned LLMs

Despite these advancements, their extensive deployment across various industries raises significant security and safety concerns, notably the potential for generating harmful, misleading, or unsafe content [4].To address these issues, techniques referred to as safety alignment have been introduced.Implemented through post-training fine-tuning, safety alignment methods like Reinforcement Learning from Human Feedback (RLHF) [5] fine-tune models to align outputs with human ethical judgments, compressing harmful responses.However, recent research has revealed significant limitations in current safety alignment methods for LLMs.First, alignment mechanisms lack robustness; even benign fine-tuning intended to enhance general performance can inadvertently weaken existing safety constraints [6].Second, despite efforts to guide models toward ethical outputs, they remain susceptible to adversarial prompts, known as jailbreaks, which bypass safety mechanisms and elicit harmful responses [4], [7], [8].Yet, crafting universally effective jailbreak prompts remains challenging, as differences in training data, model architectures, and alignment strategies severely limit their transferability, rendering existing offensive research largely ad hoc and empirical.On the other hand, recent studies have attempted to interpret the safety mechanisms in LLMs either at the layer level [9] or at the feature level [10].However, these methods may not accurately pinpoint the critical components responsible for safety behaviors as they implicate nearly 10% of model parameters as safety-related.Defensive technique [11] narrowly focuses on specific layers and is validated for limited LLMs, constraining its practical applicability across diverse/multimodal LLMs.These gaps highlight the urgent need for a deeper, principled understanding of the underlying mechanisms governing safety alignment, which could inform more targeted, reliable, and generalizable attacks.Safety Alignment as a Loophole: When analyzing the behavior of aligned LLMs, we identify an analogy between safety alignment and adversarial attacks [12]-[14], where models exhibit predictable yet abnormal responses upon receiving specially crafted inputs.The aligned models are conditioned Abstract-Safety alignment is critical for the ethical deployment of large language models (LLMs), guiding them to avoid generating harmful or unethical content.Current alignment techniques, such as supervised fine-tuning and reinforcement learning from human feedback, remain fragile and can be bypassed by carefully crafted adversarial prompts.Unfortunately, such attacks rely on trial and error, lack generalizability across models, and are constrained by scalability and reliability.This paper presents NeuroStrike, a novel and generalizable attack framework that exploits a fundamental vulnerability introduced by alignment techniques: the reliance on sparse, specialized safety neurons responsible for detecting and suppressing harmful inputs.We apply NeuroStrike to both white-box and black-box settings: In the white-box setting, NeuroStrike identifies safety neurons through feedforward activation analysis and prunes them during inference to disable safety mechanisms.In the blackbox setting, we propose the first LLM profiling attack, which leverages safety neuron transferability by training adversarial prompt generators on open-weight surrogate models and then deploying them against black-box and proprietary targets.We evaluate NeuroStrike on over 20 open-weight LLMs from major LLM developers.By removing less than 0.6% of neurons in targeted layers, NeuroStrike achieves an average attack success rate (ASR) of 76.9% using only vanilla malicious prompts.Moreover, Neurostrike generalizes to four multimodal LLMs with 100% ASR on unsafe image inputs.Safety neurons transfer effectively across architectures, raising ASR to 78.5% on 11 finetuned models and 77.7% on five distilled models.The black-box LLM profiling attack achieves an average ASR of 63.7% across five black-box models, including Google's Gemini family.