Tri-Guard: Inference-Time Abstention for Small Language Models via Logits Inspection

Small language models often generate fluent but confidently incorrect answers—especially on fact-seeking or nonsensical prompts. We present Tri-Guard, a lightweight, training-free abstention mechanism that integrates directly into Hugging Face’s transformers.generate() using standard LogitsProcessor and StoppingCriteria hooks. Tri-Guard estimates per-step risk from decoder statistics—including uncertainty (entropy, margin), stability (temperature-split JSD), and a rule-based factual critic—and triggers a controlled “grey answer” when risk exceeds a threshold. The system trades coverage for reliability, reducing harmful hallucinations without fine-tuning, retrieval, or external verifiers. We provide a fully reproducible implementation, benchmark suite, and calibration protocol, demonstrating its utility as a drop-in safety guard for low-resource deployments.