Reproduction Package for "Drift-Aware Regression Testing for Retrieval-Augmented Health Applications"

Read full text Summarise with VIRAT
Authors: Anonymouse
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

AURORA Haystack Case Study Repository This repository contains a reproducible research artifact for evaluating AURORA on a healthcare-style Haystack-ready RAG system. What is in the repo app/ — document-grounded QA system under test. aurora/ — AURORA layers: COL, SCL, CDM, RPE, RAS. scripts/ — experiment runners, table builders, and figure builders. testsuite/ — generated test catalog and schema. results/ — structured outputs for baseline, full AURORA, ablations, and bundles. state/ — snapshots, persistent cycle state, and drift fingerprints. docs/ — protocol, interpretation notes, and architecture background. Backends simple — built-in TF-IDF retrieval, fully offline. haystack — Haystack BM25 retrieval if farm-haystack is installed. Install python -m venv .venv source .venv/bin/activate pip install -r requirements.txt Optional Haystack backend: pip install -r requirements-haystack.txt Generate / refresh the test catalog python scripts/generate_tests.py python scripts/expand_tests.py python scripts/label_tests.py Experiment modes 1. static_snapshot_eval Use this for main paper tables. A snapshot freezes: corpus/docs, test catalog, backend label, model version label. All comparable configurations in one bundle read the same frozen snapshot. 2. drift_eval Use this for cycle-over-cycle evidence. This mode preserves state across cycles so CDM, RPE, RAS, and the lightweight SCL calibration signals can react to earlier outcomes. Run one clean comparable bundle This executes the following independently and stores them in non-overlapping directories: baseline full_aurora ablation_no_cdm ablation_no_rpe ablation_no_ras python scripts/run_experiment_bundle.py --backend simple --repeats 3 --experiment-mode static_snapshot_eval Results are written under per-run directories such as: results/ / cycle_01/ cycle_02/ ... Each cycle includes persisted artifacts such as: manifest.json summary.json oracle_metrics.json verdicts.json (full-suite oracle-validation verdicts when oracle evaluation is enabled) scheduled_verdicts.json (RAS-executed subset) coverage_metrics.json drift_statistics.json prioritization_metrics.json resource_metrics.json fingerprints.json prioritized_tests.json scheduled_tests.json Run drift cycles python scripts/run_aurora.py --backend simple --cycles 3 --index-version local-index-v1 This produces multi-cycle runs under results//cycle_XX/ with persisted state so you can inspect: detected drift events, impacted-test selection, ranking stability, coverage and crisk change across cycles. Run the paper-oriented index-evolution experiment python scripts/run_index_drift_experiment.py \ --backend simple \ --drift-scenario index_evolution \ --cycles 3 \ --corpus-schedule data/corpus_v1,data/corpus_v2,data/corpus_v3 \ --index-schedule index_v1,index_v2,index_v3 \ --run-id paper_index_evolution This writes per-cycle summaries plus aggregate paper-facing outputs under results/paper_index_evolution/, including: paper_cycle_summary.json and paper_cycle_summary.csv, paper_aggregate_summary.json, paper_aggregate_summary.txt, cycle-level summary.json, drift_statistics.json, and fingerprints.json. Unified CI-cycle entrypoint All experiment modes can be driven from a single CLI: python scripts/run_ci_cycle.py \ --mode aurora \ --ablation-target none \ --cycles 3 \ --budget-runtime-ms 180000 \ --budget-energy-wh 120 \ --budget-cost-usd 8 \ --model-version local-generator-v1 \ --index-version local-index-v1 \ --corpus-path data/docs \ --test-catalog testsuite/test_catalog.json \ --output-dir results \ --seed 7 Wrapper scripts are also available for the three paper-facing modes: python scripts/run_baseline.py python scripts/run_aurora.py python scripts/run_ablation.py --ablation-target cdm python scripts/run_experiment.py Oracle validation against ground truth The repository now supports a paper-facing validation pass for the Cognitive Oracle Layer (COL) using testsuite/oracle_ground_truth.json. Ground-truth records are deterministic catalog-derived labels rather than post-hoc human execution annotations. The ground-truth file is automatically regenerated from testsuite/test_catalog.json whenever oracle evaluation runs, so the dataset cannot silently go missing or drift out of sync with the catalog. A sample is labeled pass when the catalog expects a grounded answer with a reference_answer. A sample is labeled fail when the catalog encodes abstention / clarification behavior (fallback, asks_for_clarification) or does not define a reference_answer. Run it directly on an existing full-suite verdict snapshot: python scripts/evaluate_oracle.py \ --test-results results//cycle_01/verdicts.json \ --ground-truth testsuite/oracle_ground_truth.json \ --output results//oracle_ground_truth_metrics.json Or enable it as part of a run (enabled by default; pass --evaluate-oracle false to disable it): python scripts/run_aurora.py \ --backend simple \ --cycles 3 \ --evaluate-oracle true When enabled, the aggregate validation output is written to: results//oracle_ground_truth_metrics.json The output includes: agreement_rate false_accept_rate false_reject_rate evaluation_scope (all_tests for the default pipeline-integrated sweep) available_verdict_count optional calibration buckets when verdict confidence is available Paper-ready tables and figures python scripts/build_paper_tables.py python scripts/build_paper_figures.py Outputs: results/paper_tables/main_results.csv results/paper_tables/ablation_results.csv results/paper_tables/cycle_results.csv results/paper_figures/runtime_vs_crisk.png results/paper_figures/cost_vs_failures.png results/paper_figures/ttfcd_comparison.png results/paper_figures/ablation_bar_chart.png Component behavior COL: semantic + evaluator + metamorphic verdicts. SCL: prompt-behavior coverage plus lightweight coverage/failure calibration for future ranking. CDM: fingerprints docs and model label, reports actual detected drift events, and narrows revalidation candidates. RPE: ranks candidates using risk, drift exposure, historical failures, and coverage signal. RAS: schedules tests under runtime/energy/cost budgets and records why tests were selected. Citation-preserved definition A response counts as citation-preserved when: it contains a parseable Sources: block, and at least one cited item matches retrieved evidence for that answer. This keeps evaluation strict about grounding while avoiding false failures caused by harmless citation formatting variation. Reproducibility notes Use static_snapshot_eval for baseline and ablation comparisons. Reproduce results with GitHub Actions You can reproduce the repository results directly from the Actions tab without a local Python setup. Option A: full end-to-end paper pipeline Go to GitHub → Actions → AURORA End-to-End Pipeline. Click Run workflow. Choose inputs: backend: simple (default, fully offline-friendly) or haystack aurora_cycles: number of CI cycles (default 3) keep_existing_catalog: set true to reuse committed testsuite/test_catalog.json Run the workflow and wait for completion. Download the workflow artifact(s) to inspect generated outputs under results/ and state/. This workflow executes indexing, test generation/expansion/labeling (unless keep_existing_catalog=true), static-snapshot experiment bundle, drift cycles, and paper table/figure builders. Option B: targeted CI experiment (baseline / aurora / ablation) Use GitHub → Actions → Run CI Experiment when you want a single mode run with explicit budgets and labels. Key inputs: mode: baseline, aurora, or ablation ablation_target: cdm, rpe, or ras (when mode=ablation) cycles, budget_runtime_ms, budget_energy_wh, budget_cost_usd model_version, index_version, corpus_path, test_catalog, seed, backend Outputs are uploaded as artifacts containing results/ and state/. Option C: paper index-evolution drift experiment Use GitHub → Actions → Run Index Drift Experiment for the explicit index-drift scenario used by the paper-oriented analysis. Set corpus_schedule, index_schedule, cycles, and run_id; the workflow uploads results/ and state/ as artifacts. Do not compare runs with different snapshot IDs in the same ablation table. drift_eval is for demonstrating CDM/SCL behavior over time, not for the main static ablation table. If a component effect is small in your environment, prefer conservative interpretation in the paper text.

Fields of Study & Subjects

Software Testing and Debugging Techniques Software Engineering Research Adversarial Robustness in Machine Learning

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
27 March 2026
Language
EN