Counterfactual Invariance Tracing: Per-Claim Epistemic Confidence from Model Internal Representations

Read full text Summarise with VIRAT
Authors: Frederic David Blum
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Counterfactual Invariance Tracing: Per-Claim Epistemic Confidence from Model Internal Representations Authors Frédéric David Blum · Catalyst AI Research · Tel Aviv, Israel ORCID: 0009-0009-2487-2974 Description We introduce Counterfactual Invariance Tracing (CIT), a novel algorithm that produces per-claim epistemic confidence scores from a single augmented forward pass through a large language model. CIT addresses the critical unsolved problem in enterprise LLM deployment: no existing method reliably distinguishes between claims grounded in learned knowledge and claims fabricated through interpolation in latent space. The Problem Current approaches — logprob thresholding, self-consistency sampling, retrieval-augmented generation, and trained verifiers — each probe a single dimension of the generation process and fail on specific, well-characterized adversarial cases. A claim can be stable in logprobs yet fragile under rephrasing. It can be consistent across samples yet brittle under perturbation. The Algorithm CIT triangulates three complementary probes of the model's internal representations: Stability: Measures the coherence of a claim's activation trajectory across layers. A grounded claim develops smoothly through the network; a fabrication may exhibit abrupt directional changes. Resilience: Measures robustness under semantically neutral perturbations to hidden states. A grounded claim occupies a stable basin in representation space; a fabrication sits in a fragile region. Invariance: Measures preservation of the claim's representation under meaning-preserving input transformations. A grounded claim is encoded abstractly; a fabrication may depend on surface-level patterns. The composite confidence score is computed as a weighted geometric mean: c_k = (s_k^α · r_k^β · i_k^γ)^{1/(α+β+γ)} The geometric mean enforces that a low score on any single component pulls the composite down multiplicatively, implementing the triangulation principle. Theoretical Foundation We formalize three propositions establishing that no single-probe method suffices, and state the Triangulation Conjecture: the probability that a hallucinated claim simultaneously exhibits high stability, high resilience, and high invariance decreases exponentially with model capacity. The three components admit a unified interpretation in terms of information geometry: stability measures geodesic coherence on the Fisher manifold of representations, resilience measures local curvature, and invariance measures the intrinsic dimensionality of the claim's representational subspace. Key Properties Per-claim: Confidence is assigned to individual factual assertions, not entire sequences Domain-agnostic: No domain-specific training required Efficient: Designed for ≤20% computational overhead over standard inference Testable: Complete benchmark protocol defined with explicit success criteria (AUROC ≥ 0.87 across 5 domains) Benchmark Protocol The paper defines a rigorous experimental protocol including: Standard benchmarks (TruthfulQA, MedConCite) A proposed Coherent Hallucinations test suite for hard cases Systematic ablation design across all 7 component combinations Explicit baselines and success thresholds Failure Modes (Honest Assessment) CIT is a relative confidence signal, not an absolute guarantee. It audits the model's internal consistency of belief, not its correspondence to ground truth. Systemic misconceptions deeply embedded in training data will receive high confidence scores. The algorithm requires access to model internals (hidden states, gradients) and cannot be applied to closed API-only models without approximation. Defensive Publication Notice This paper is published as a defensive disclosure to establish prior art. The theoretical framework, algorithm definition, mathematical foundations, and benchmark protocol are fully specified herein. This publication is intended to prevent any party from obtaining patent protection on the CIT algorithm or its described components. The detailed reference implementation will be released separately. Related Work Catalyst AI Research: catalystais.com CST / FDBC Programme: DOI 10.5281/zenodo.18859602 SDSQ Programme: DOI 10.5281/zenodo.18779189 Cortex Protocol: DOI 10.5281/zenodo.19003627 Sigil Protocol: DOI 10.5281/zenodo.19027462 Keywords hallucination detection, epistemic confidence, language model interpretability, per-claim verification, counterfactual analysis, information geometry, defensive publication License Creative Commons Attribution 4.0 International (CC BY 4.0) Upload Type Publication — Preprint Communities Machine Learning, Artificial Intelligence, Natural Language Processing

Fields of Study & Subjects

Adversarial Robustness in Machine Learning Topic Modeling Explainable Artificial Intelligence (XAI)

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
22 March 2026
Language
EN
Document type
Preprint