AOI v4 Domain-Agnostic Moat: 28-Domain Cross-Domain Validation Evidence
Abstract
Cross-domain validation evidence for AOI v4 (DOI: 10.5281/zenodo.21368154), the numpy-only confidence-gated classifier. The same untouched algorithm — no retraining, no domain-specific tuning — was run against 28 independent real public datasets spanning finance, medicine, agriculture, ecology, forensics, security, engineering, wearables, real estate, and social science, each at a fixed 85% coverage target. Results: the confidence gate produced a real accuracy lift on every domain where an ungated comparison was run (10 of 28), typically +2 to +5 percentage points, never approaching the flagship ~97%→99.6% lift on the original Breast Cancer Wisconsin benchmark. Two results are flagged as likely inflated by feature leakage (student performance via prior-period grades; bank marketing via call-duration). Three results are honestly weak or near the majority-class baseline (contraceptive method choice, forest fire burned area, and a severely imbalanced yeast protein localization task) and are reported as-is, not excluded. This record documents the domain-agnostic generalization claim with unmanipulated, reproducible evidence: no result was cherry-picked, re-run for a better number, or hidden. Full per-domain reports, exact commands, and verbatim tool output are included. Correction (2026-07-29): The 28 headline accuracies in this record are in-sample, not held-out. The command recorded for every domain fits GatedModel on all rows and calls evaluate(X, y) on those same rows, so the published gated figures are training-set figures; they reproduce exactly, but several are materially optimistic against a stratified 70/30 held-out re-run over 10 seeds - Sonar 98.30% in-sample against 81.45% +/- 4.25 held-out (16.85 points), Glass 74.73% against 67.07% +/- 6.04, Ionosphere 96.98% against 92.81% +/- 2.30, Forest fires 50.57% against 46.49% +/- 2.79, and Heart disease (Cleveland) 89.68% against 86.03% +/- 2.72. The gate effect itself survives the correction: the ungated comparison now exists for 28 of 28 domains rather than 10 of 28, the held-out gated-minus-ungated delta is positive on all 28 domains with every 95% confidence interval excluding zero (range +0.60 to +5.87 points, median +3.83), and a matched random-abstention control that drops the same number of rows gives a mean delta of +0.106 points. Three domains are nevertheless withdrawn as gate evidence: Student performance is a measurement of the leaked prior-grade columns G1 and G2 (removing them takes held-out gated accuracy from 95.94% to 70.83% against a 67.09% majority baseline, and the published 99.40% is not a domain-generalization result); Bank marketing's +5.09 delta is driven by an 88.73% majority class rather than by discrimination (held-out ungated accuracy without the duration column is 89.96%, a lead of 1.23 points over always answering "no"); and Forest fires is a failure rather than a weak result, at 46.49% held-out against a 47.78% majority baseline. Room occupancy carries a related caveat: Light alone accounts for about 10 points, and removing it moves held-out gated accuracy from 99.54% to 89.29% while the gate delta rises from +0.60 to +6.90. One further clarification, because the record's framing invites the confusion: the confidence gate measured here is the max-class probability of a plain 800-iteration gradient-descent logistic model, not the 96D octonion collapse engine. Benchmarked on the identical 28 domains, 10 held-out seeds and matched 85% coverage, the collapse engine's unified_confidence gates at chance (AUC 0.483, 95% CI 0.459-0.507), loses to softmax gating on 28 of 28 domains (mean head-to-head +3.76 points in softmax's favour), and loses to the L2 norm of its own 24-dimensional input (+0.60 points against -0.14). Repairing the engine's discontinuous qubo_quality branch removes a genuine reproducibility defect - cross-realization correlation moves from -0.629, negative on 28/28 domains, to +0.831 - but does not change the benchmark outcome (repaired variants: 0 of 28 domains beating softmax, gate AUC 0.483 to 0.511). The defensible version of this record's claim is therefore: a confidence gate at 85% coverage produces a small, consistent and statistically reliable held-out accuracy gain of +0.6 to +5.9 points (median +3.8) across 28 independent public datasets, verified against a random-abstention control, on 25 of the 28 domains.
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 20 July 2026