Supplementary Material: Auditing a Wastewater Anomaly Detection Benchmark Before Model Selection

Auditing a rule-labelled anomaly-detection benchmark before model selection Supplementary material for Auditing a Wastewater Anomaly Detection Benchmark Before Model Selection: Label Recovery, Feature Dimensionality and Comparison-Window Integrity, submitted to Journal of Hydroinformatics as HYDRO-D-26-00114. DOI: 10.5281/zenodo.22094391 — the version DOI for release 2.0.0, the release the accompanying paper reports. Licence: MIT for code and notebooks; see LICENSE-DATA.md for the data, which is not MIT and not uniformly CC-BY. If the archive was downloaded as zip files, unpack them all in one place before running anything; the paths below assume the unpacked tree, with data/, notebooks/, results/ and scripts/ side by side. What the work does Anomaly-detection benchmarks are often labelled by a rule applied to the same measurements a model is then given as inputs. When that happens, a model that scores well has recovered the rule rather than learned the process, and nothing in a conventional evaluation says so. This archive contains a set of checks that test for it before any model is fitted, and a worked application to one such benchmark. The checks that decided something here need no model at all — they are the first three steps below, and they run in 0.023 s on all 2,000 records. A gated selection protocol built on attribution was also tested, and it is reported as a negative result: on this cohort it admitted one candidate of eighteen under an interval reading, its attribution gate rejected nothing, and every system cheaper than alarming unconditionally — 221, 226 and 229 against 249 — carries precision below 0.41 against a charter floor of 0.80, so the charter excludes the entire cost-minimal region by construction. The gates are archived here because the finding is about them: Gate Question 1 — metric charter does the model perform well enough to be worth inspecting? 2 — attribution reliability is its reliance spread, and stable under resampling? 3 — complementarity do two survivors actually cover each other's weaknesses? head-to-head is a merged system worth its complexity? Each gate returns its decision and the table it decided from, so a rejection can be inspected rather than taken on faith. Declining to build a hybrid is a valid terminal state. Principal findings on the benchmark analysed The label is a box rule on twelve raw parameters, reproduced at 100.0000%. 527 of 777 positives (67.8%) violate a constraint the model observes the same day; 776 counts a 55-day-lagged proxy as access and is an upper bound. A depth-1 decision tree on one variable reaches precision 1.0000. A depth-3 tree prints the rule's own thresholds — 30.05, 100.00, 799.70. Lagging every predictor by one day drops AUC-PR from 0.90 to 0.44; skill above the prevalence floor falls from 0.844 to 0.125. Most of the apparent performance was rule reconstruction. COD_Conc = (COD_in/Q_in)·1000 is dimensionally invalid and 61.1% explained by 1/Q_in against 37.5% by COD_in, on the training partition. (The pipeline notebook computes this over the whole 1,945-row modelling frame and prints 63.5% against 37.4%. The audit is a pre-fit check, so it belongs on training data; scripts/recompute_ratio_tables.py is the generator of record for the paper's value, and the notebook's own output is left as it ran.) Removing it and refitting costs at most 0.0162 AUC-PR, and all three paired intervals contain zero — high attributed importance, dependence not resolved by this test set. Across the full sweep of thirty-six refits only the top-ranked feature is established as mattering; the ordering below it is not. A distribution-shift signal of p = 5.89 × 10⁻¹² becomes p = 0.322 when the comparison windows are matched on month. 0 of 20 features differ under matched windows. The 55-day lag that produced the signal was introduced by a search on this record, so the finding is a self-audit. A decision threshold is a fitted parameter, and re-deriving one after the refit leaks. Three of the four representation paths refit on train+validation once thresholds are fixed. Re-deriving the gradient-boosted model's cost-minimising threshold on validation after that refit gives a test cost of 177, against 229 for the lowest any model reaches when its threshold comes from the train-only fit — and within four units of that model's test-set optimum of 173, which no honest procedure can see. The 52-unit gap is the size of the leak, not a better system. Both are in results/tables_final/architecture_comparison.csv, where the leaked rows are labelled (in-sample, post-refit); §12.1 of the pipeline notebook does the same search leakage-free over every model. The label-free checks are then applied to UCI dataset 106, an operating plant's records these authors did not produce. The dispersion criterion returned the opposite verdict there and was correct on both records — though the closed form shows it could not have failed on either, so what this establishes is that a criterion fixed in advance separated them, not that dispersion predicts dependence. Whether UCI 106 retains demonstrable shifts under a properly specified dependence-aware test is open, so the window procedure's ability to preserve real signal is not established here. Contents notebooks/ 6 two families, each Template / FINAL / Analysis scripts/ 52 the gates module, the builders, the audit, the recompute scripts, the section sources data/ 6 the benchmark, and UCI 106 with its dates preserved database/ 4 the Optuna study databases the notebook loads instead of re-tuning results/ 148 99 tables and 33 figures for the main record, 16 for the external one Not everything in results/ has generating code here, and pretending otherwise would be the wrong kind of claim for this paper to make. The pipeline notebook writes most of it. Eighteen of the ninety-nine tables were produced by ad-hoc analysis passes that are not part of this archive. REPRODUCE.md §7 lists them and says which claims they carry. Eleven more, and both figures that had been shipped as published, now have a generator of record --- see the table at the end of §7; each was verified against its deposited copy before its generator was accepted, and scripts/recompute_record_tables.py --check re-runs that comparison. Read these first, in this order: File What it answers DATA-PROVENANCE.md what the two records are, and what is and is not known about each DATA-DICTIONARY.md every column, its unit, its documented interval, its missingness — and the two fields the source does not supply REPRODUCE.md the end-to-end workflow, and what is not reproducible MANIFEST.md a SHA-256 for every file in the archive environment.json Python, platform, hardware, seed, trial budget, every package version Each family is a triad sharing the same cells and the same §S.N identifiers, so a section can be read across all three by the same name: Template — the specification. Every code cell holds a WHAT / HOW / WHY / VERIFY comment and no implementation. FINAL — the same cells with working code, executed. 95 / 95 pass in ~230 s. Analysis — the same cells again, each carrying its captured output and the interpretation. Outputs are read from the executed FINAL at build time, so the written analysis cannot drift from what ran. The validation family is 12 code cells and runs in about four seconds. Running it REPRODUCE.md is the end-to-end workflow, including how to verify what you regenerate against what we got, and a plain statement of the three things that are not reproducible from this archive. The short form: pip install -r requirements.txt The cheapest check, and the paper's central claim — no model, no tuning, no fitting, and it either reproduces the supplied label exactly or it does not: python scripts/verify_label_rule.py python scripts/nb_exec.py "notebooks/Final Anomaly Detection Pipeline FINAL.ipynb" The validation notebook needs nothing from this archive except itself. It downloads UCI 106, records the file's SHA-256, and proceeds: python scripts/nb_exec.py "notebooks/External Validation UCI FINAL.ipynb" environment.json pins the Python version, platform, seed, trial count and the dataset's MD5. MANIFEST.md gives a SHA-256 for every file here. Rebuilding the notebooks They are build artefacts. Code lives in scripts/code_sections/, interpretation in scripts/analysis_sections/, both keyed by cell id: python scripts/build_final_nb.py && python scripts/build_analysis_nb.py Add --family validation for the second family. scripts/audit_notebooks.py runs ten classes of mechanical consistency check across both — cell coverage, cross-reference validity, unseeded randomness, leakage, and whether every number in the written analysis appears in an output. Applying the protocol to other data The gates take arrays and frames, not this study's globals: from pipeline_gates import run_pipeline report = run_pipeline(results, importances, shap_values, feature_names, val_scores, test_scores, y_val, y_test) Individual checks are importable on their own — nesting_check, correctness_mcnemar, matched_window, attribution_stability, attribution_concentration. Data The record is in this archive, at data/WWTP_data_2019-2024.csv, in full and unmodified, so every claim in the paper can be checked against it. It is third-party material whose origin and labelling the provider does not document, and which carries no licence and no identifiable rights holder; no licence is asserted over it here, and any rights holder is invited to contact us. data/THE-RECORD.md carries the digests and the reasoning. Do not use it as an anomaly-detection benchmark. It does not behave like plant instrumentation — that is the paper's central finding, not an aside — and a performance figure computed on it is not evidence about any treatment works. Read DATA-PROVENANCE.md before using it fo