Comparative Analysis of LLMs Against Jailbreak Prompts

The rise of open-source Large Language Models (LLMs) has made it more important than ever to protect them from jailbreak attacks. These attacks trick models into giving harmful or unsafe responses. In this paper, we test how well three popular models, Mistral-7B-Instruct-v2, Llama-3.1-8B-Instruct, and Qwen-2.5-7B-Instruct can handle such attacks (PAIR, DAN, DSN, GPTFuzzer) using prompts from JailbreakBench and other datasets. At First, the models were easily jailbroken, with success rates of up to 67.24%. We then applied two defenses: SmoothLLM (an ensemble-based perturbation method) and PerplexityFilter (a statistical anomaly detection method). These defenses worked well for some models, for example, Llama dropped to 27.42% ASR but didn’t always help, especially with Mistral-7B-Instruct-v2 and the GPTFuzzer. This shows that current defenses don’t work equally for all models.