A11Y.md Efficacy Benchmark — Study 3: The v2.0.0 Audit (raw data)

Read full text
Authors: Felipe Carriço
Cited by: 1

Abstract

Complete raw record of Study 3 of the A11Y.md benchmark — the v2.0.0 audit — preregistered at osf.io/wt5n4 and published unconditionally on outcome, as registered. The study compares A11Y.md v2.0.0 against v1.8.0 (the version measured by Studies 1–2) on a frozen 7-screen bookstore-journey unit: 4 conditions (no guidance; a generic "Make it accessible." phrase; the standard at git tag v1.8.0; at tag v2.0.0 including its contrast tool) × 2 agents (Claude Code 2.1.237; Antigravity 1.1.26) × 5 runs = 40 collected sessions, 39 retained. Primary contrast: v2.0.0 vs v1.8.0, same agent, never pooled. Files: a11ymd-study3-screens.zip — every generated screen and asset, per run · a11ymd-study3-raw.zip — the clients' raw session captures · a11ymd-study3-verify.zip — pinned axe-core 4.13.0 verification (JSONL per run) and full-page screenshots from the same pass · a11ymd-study3-analysis.zip — the frozen registered analysis output and the append-only collection log (dual records preserved) · a11ymd-study3-quarantines.zip — every quarantined false start and harness episode, nothing deleted · a11ymd-study3-probes-pilot.zip — environment gate probes and the discarded pre-freeze pilots · a11ymd-study3-human-qa.zip — the human-verification pass (seeded blind sample, sealed map, checklist, verdicts) · MANIFEST-study3.json — MD5 of every file. Headline results, the dated deviations journal (including a 10-lens pre-publication audit), and all hash-frozen instruments live in the project repository: github.com/fecarrico/A11Y.md (benchmark/round2/). Raw data from Studies 1–2 is archived separately at 10.5281/zenodo.22073025.

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
6 September 2026
Language
EN
Document type
Dataset