Evaluation Protocols for AI-Enabled Systematic Review

Prizmatic is a research project developing and evaluating methods to speed up medical evidence synthesis through systematic reviews while keeping the process auditable and aligned with established systematic review standards. We are building a tool-agnostic benchmark that asks AI-aided systematic review systems to reproduce core outputs of human led systematic reviews of randomized controlled trials, using only the inputs a reviewer would typically report (review question, PICO criteria, and published search strings). This evaluation protocol examines whether these AI systems can reliably recover the same included studies as the human-led systematic review (HL-SR), and whether they can accurately extract the key outcome values used in the analysis. We track performance stage by stage (retrieval, screening, full text eligibility, inclusion, extraction), quantify accuracy with standard metrics, and measure reproducibility by repeating runs under a frozen configuration. All protocol versions, run manifests, and reference datasets are archived to support transparency, comparison across tools, and future expansion to a larger, reusable benchmark dataset across multiple clinical domains.