audit-closed-ai-scientist

Audit-Closed AI Scientist Benchmark Description This repository provides a reproducible benchmark for evaluating the statistical validity of AI scientist systems and autonomous research agents. Recent advances in automated scientific discovery and *AI scientist* architectures suggest that machine learning systems may autonomously generate hypotheses, design experiments, and evaluate results. While such systems enable large-scale exploration of scientific hypotheses, adaptive experimentation also introduces well-known statistical risks, including: - p-hacking - optional stopping - hypothesis shopping - adversarial experiment submission Without appropriate safeguards, these behaviors can inflate false discoveries and undermine the reliability of automated research pipelines. This benchmark provides a controlled experimental environment that reproduces these failure modes and evaluates an alternative governance mechanism based on an **Audit-Closed AI Scientist protocol**. The central principle is that acceptance decisions must depend only on a publicly verifiable experiment log: Accept_t = f(Log_0:t) All hypotheses, experiments, and evaluations are recorded in a transparency log. Evidence accumulation is implemented using sequential e-process inference, which remains valid under continuous monitoring and adaptive experimentation. The repository includes: - baseline simulations of naive AI scientist pipelines - experiments demonstrating statistical failure modes - an implementation of the Audit-Closed protocol with transparency logs and deterministic replay - sequential inference using e-process statistics - reproducible benchmarks comparing naive and audit-closed research agents The goal of this project is not to implement a complete autonomous scientist, but to provide a **benchmark for evaluating the statistical reliability of autonomous scientific discovery systems**. Reproducibility All simulations and benchmarks are fully reproducible using the provided scripts. Figures and experiment results can be regenerated directly from the repository. Intended Audience This benchmark may be useful for researchers working on: - AI scientist architectures - autonomous research agents - automated scientific discovery - self-driving laboratories - scientific machine learning infrastructure Keywords AI scientist, autonomous research agents, automated scientific discovery, reproducible benchmark, sequential inference, e-process statistics, transparency log, self-driving laboratories, reproducible science