The Vyapti Probe Benchmark: A Navya-Nyaya Framework for Diagnosing Logical Reasoning Failures in Large Language Models

Abstract This paper introduces the Vyapti Probe Benchmark, a formally verified 100-problem evaluation suite designed to test whether large language models distinguish statistical regularity from vyāpti (invariable concomitance)—the foundational inferential concept in Navya-Nyāya epistemology. The benchmark comprises 50 probe problems containing genuine vyāpti violations paired with 50 matched controls where universal rules hold, spanning five HETVABHASA (logical fallacy) categories from classical Indian logic. Methodology: Each problem includes Z3 SMT-solver verification for machine-checkable ground truth. Evaluation uses a five-tier protocol assessing answer correctness, structural adherence to six-phase Nyāya reasoning, vyāpti explicitness, formal verification, and fallacy classification. Six model configurations were evaluated: two base models, two Pramana fine-tuned models, and two prompting controls. Key Findings: No significant probe-control gap in base models (35% vs 36%; p=0.926), though larger samples needed for definitive claims Form-substance split: Pramana Stage 1 achieves 96% structural adherence to Nyāya methodology while scoring 4pp below base model on probe correctness (24% vs 28%) Complete diagnostic coverage: All 416 model failures map to HETVABHASA categories with 77.4% assisted predictive accuracy (66.7% strict) Resources: Dataset: https://huggingface.co/datasets/qbz506/vyapti-probe-benchmark Code: https://github.com/TechNektar/pramana/tree/main/data/vyapti_probe Evaluation Notebook: https://colab.research.google.com/github/SharathSPhD/pramana/blob/main/notebooks/03_vyapti_evaluation.ipynb The work demonstrates that entropy minimization discovers regularities without establishing vyāpti, validating Navya-Nyāya's 2,500-year-old taxonomy as a principled diagnostic framework for contemporary AI reasoning failures.