A benchmark for automated scientific manuscript evaluation using retracted publications

Read full text Summarise with VIRAT
Authors: Andrew Michael Brilliant
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Pre-registration concept note for a benchmark evaluating automated scientific manuscript assessment systems. The benchmark uses retracted publications as ground truth, partitioned by model training cutoff dates to distinguish memorization from reasoning. Five evaluation conditions are tested: single model baseline, naive ensemble, structured anthropomorphized agents (single model), multi-model structured agents with calibrated weighting, and an exploratory human-curated model-role assignment condition. No results are reported. This document establishes the experimental design prior to execution. Keywords: benchmark, manuscript evaluation, retracted publications, machine learning, pre-registration, LLM, multi-model inference License: Creative Commons Attribution 4.0 International (should already be selected)

Fields of Study & Subjects

Academic Writing and Publishing scientometrics and bibliometrics research Academic Publishing and Open Access

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN
Document type
Preprint