Measuring susceptibility: A benchmark for conspiracy theory adherence in large language models

A critical vulnerability exists within state-of-the-art large language models: while robustly debunking scientifically baseless claims like the “Flat Earth Theory” they consistently fail to reject politically plausible conspiracies that mimic legitimate discourse. We term this the “plausibility gap”. To systematically quantify this risk, we introduce the Conspiracy Adherence Score (CAS), a novel risk-weighted metric, and present the first large-scale benchmark of this phenomenon. Analyzing over 28,500 responses from 19 leading LLMs, our results reveal a stark hierarchy of failure. Model adherence to Level 1 theories rooted in real-world political concepts (e.g., “Active Measures” “Psyops”) was, on average, over five times higher than for more moderate (Level 2) theories. Performance varied dramatically across models, from one achieving a perfect score via a 100 percent refusal strategy to others assigning significant credibility to harmful narratives. This demonstrates that current AI safety measures are brittle, optimized for simple factual inaccuracies but unprepared for narrative warfare. Without urgent intervention, LLMs risk becoming authoritative vectors that launder politically charged disinformation under a veneer of neutrality. Our benchmark provides the first diagnostic tool to measure and mitigate this specific, high-stakes failure mode.