Pattern-Matching Failures in LLM Refusal Detection: A Case Study in Detector Reliability

Pattern-based refusal detection — classifying a model’s response to an adversarial prompt as a refusal, a compliance, or an uncertain outcome via lexical matching — is common in LLM safety and red-teaming pipelines. We report a case study in which an apparent “non-determinism” finding, where Claude Haiku produced inconsistent UNCERTAIN verdicts across comparable adversarial prompts, was substantially an artifact of our own open-source evaluation framework’s detector rather than of the model. We identify two concrete defects in safelabs-eval’s refusal-matching logic: a Unicode curly-apostrophe vs. ASCII-apostrophe mismatch, and an incomplete hardcoded refusal vocabulary. Because raw responses from our original exploratory run were not preserved, we report a controlled replication instead: applying the old (v0.2.0) and fixed (v0.2.1) detector to one independently saved 30-prompt Claude Haiku response set, holding responses constant and varying only the detector version. On this comparison, the UNCERTAIN rate drops from 14/30 to 8/30, and all 8 remaining cases trace to specific, identifiable vocabulary gaps rather than genuine model ambiguity. We report this replication figure — not the differentlyderived count from our original, unpreserved exploratory run — as the paper’s primary result, and we disclose what that provenance gap itself implies about raw-data retention practice in red-teaming pipelines generally. We argue the core finding is not an isolated bug report but a symptom of a broader, underscrutinized reliability problem in pattern-based safety evaluation, and we outline what an evaluation framework should report about its own detector behavior — not just about the model it is testing.