ReviewAid: AI-Driven Full-Text Screening and Data Extraction for Systematic Reviews and Evidence Synthesis

Release Notes - ReviewAid v4.0.0 Why this release exists The v3.0.0 architecture validation against human gold standards (CSMeD-FT full-text screening, ~2,000 papers; EvidenceInference extraction, ~2,184 papers) proved the safety story and falsified the automation story: The Tier-1 keyword gate was the largest error source: 133 auto-exclusions per backend, 53 of them killing papers the review teams had included — each delivered with confidence 1.0 and zero evidence (men matching inside women, adults firing on any paper, acute LBP matching in a chronic-pain trial's introduction). Single-shot screening was unstable: one prompt decided everything, local models frequently returned unusable output that silently became regex-fallback decisions (up to 91% of one arm), and identical runs could flip decisions. Effect-direction extraction scored at chance (36.3% on a 3-class task). Confidence was a self-reported scalar — honest after deterministic repair (ECE 0.43–0.66 → 0.19–0.23) but not actionable. v4.0.0 fixes the gate, rebuilds the LLM stage around evidence and agreement, and makes every decision auditable — without changing the architecture: Tier-1 deterministic gate → Tier-2 LLM screening → deterministic confidence override, exactly as published. What changed, how, and why 1. Tier-1 keyword gate rebuilt — tier1_gate.py (new module) How: a keyword hit now qualifies only after deterministic guards: Word-boundary matching — men no longer matches women, rat no longer matches ratio. Negation guard — "no acute LBP", "non-pregnant", "free of…" don't fire (vocabulary extends the Check E negators already in confidence.py). Background-context guard — mentions in background discourse (previous, studies have, had shown, although…) are ignored unless eligibility language (enrolled, randomised, criteria…) is nearby. Corroboration — a lone one-word criterion (adults, children) can never auto-exclude; it needs a second qualifying criterion. One new entry point, evaluate_tier1(), returns the decision, qualified criteria, and every discarded hit with its reason (negated / background) — so every exclusion stays auditable. find_exclusion_matches keeps its name and signature as a compatibility wrapper. Measured on the full deposited corpus (1,968 papers): Auto-exclusions: 133 → 44 — a strict subset of v3.0.0's. Gold-include papers killed: 53 → 17 (−68%). New auto-exclusions v3.0.0 would not have made: 0 — locked by property tests. Further tightenings were prototyped, measured, and rejected (documented in CHANGELOG) — the gate is at its deterministic ceiling, and we can prove it. 2. Tier-2 screening rebuilt — pico_screen.py (new module) The single holistic prompt is gone. In its place: Per-criterion judgments with evidence: the model judges each criterion separately (yes/no/unsure) and must cite the supporting sentence. Every quote is verified against the paper text — an unfindable quote downgrades the judgment to unsure (grounding, applied to screening). Three API calls per paper, flat: three independent judgments, majority-voted per criterion (self-consistency; Wang et al. 2022). The sample agreement rate replaces the self-reported confidence scalar — confidence is now a measured property, not a feeling. Three calls is the accuracy-per-cost sweet spot: voting gains are front-loaded, and the vote absorbs single-sample errors and model refusals. Recall-first exclusion: exclusion fires only on exclusion evidence that is unanimous across samples and grounded in a quote. Split votes and quoteless verdicts refer — weak evidence can never remove a study. Tiebreaker adjudication: when samples split on a criterion, one senior-reviewer call settles it against the competing quotes, grounded like every other judgment (one extra call, only on disagreement). Priority queue score: every paper leaves with a priority (inclusion strength, quote coverage, agreement) so the human review queue is worked highest-first — workload-saved at fixed recall is now measurable straight from the export. Honest failure: if all samples fail, the paper is a Maybe referral — never a silent regex-fallback exclusion. Temperature 0.0; per-paper criteria_trail (verdict, agreement, quote) in every result; System Terminal reports stage, usable samples, agreement, and discarded keyword hits. 3. Extraction: structured effect direction Effect Direction is now a closed label set (significantly increases / significantly decreases / no significant difference / unclear) plus an Effect Direction Evidence field that must quote the paper. Strict scoring becomes possible; the label's confidence is verified through its evidence sentence. No extra API calls. 4. Reliability and infrastructure Ollama context-window fix (utils.py): Ollama was silently running a ~2k-token context, truncating long prompts mid-contract — a major hidden cause of fallback cascades on full-text screening. Now num_ctx = 16384 by default, tunable via OLLAMA_NUM_CTX. parser.fallback_uses(): counts regex-fallback decisions so a validation pilot can fail any arm whose fallback rate is too high. Prompt wording hardened against local-model safety refusals; screener batch limit 20 → 10 (extractor unchanged; local clones unrestricted); screening criteria inputs now guide users toward sentence-style criteria — one-word criteria were v3's root cause. 5. Tests and documentation 81 tests (up from ~20): guard-by-guard gate tests, property tests (the gate can only defer, never expand), screening-engine tests, table-type regression tests, version-sync test. CI runs all. README Tier-1/Tier-2 sections rewritten; CHANGELOG.md and ROADMAP.md document what changed, what was measured, and what was rejected. {If anyone reads this, thanks for using ReviewAid, and you're awesome.} What v4.0.0 delivers vs v3.0.0 | Metric | v3.0.0 (measured) | v4.0.0 | |---|---|---| | Fatal confident gate errors | 53/backend | 17 (−68%) — measured | | Gate auto-exclusions | 133 | 44 — measured | | Screening sensitivity | 36.0–85.5% | +1.8–4.4pp gate layer; +4–8pp projected with the pipeline | | Screening specificity | 12.3–55.1% | −0–4pp (the deliberate recall-first trade) | | Unusable/hidden failures | up to 91% of an arm | 0 silent — referrals + fallback_uses() | | Confidence | self-reported scalar | measured agreement + deterministic override (unchanged) | | Auditability | reason text only | per-criterion verdicts + quotes + discarded-hit log + priority | | Effect direction | at chance (36.3%) | strictly scoreable labels + evidence quotes | | API calls per screened paper | 1 | 3 | | Run-to-run stability | flips observed | temperature 0 + agreement measurement | Known limits (unchanged, stated plainly) Effect-direction accuracy is still model-bound; over-inclusion remains the LLM tier's weakness; criteria quality is now the largest single lever (garbage criteria in, referrals out); full-text screening is not fully automated by this or any published system. ReviewAid remains what its paper claims: a third reference under human oversight — v4.0.0 makes that reference harder to fool and easier to check. What's Changed modifications (partial for v4.0.0) by @aurumz-rgb in https://github.com/aurumz-rgb/ReviewAid/pull/71 v4.0.0-testing by @aurumz-rgb in https://github.com/aurumz-rgb/ReviewAid/pull/72 v4.0.0 ReviewAid by @aurumz-rgb in https://github.com/aurumz-rgb/ReviewAid/pull/73 Full Changelog: https://github.com/aurumz-rgb/ReviewAid/compare/v3.0.0...v4.0.0