AI as gatekeeper: a cross-sectional study of AI-generated peer review of landmark oncological studies

The increasing volume of scientific manuscripts has strained traditional peer-review systems, prompting interest in artificial intelligence (AI) as a potential solution. However, the comparative performance of large language models (LLMs) in critically appraising oncological research remains unexplored. We conducted a cross-sectional study evaluating the peer-review capabilities of three LLMs—Claude Sonnet 4.5, Google Gemini 3.0 Thinking, and OpenAI ChatGPT 5.1—using 50 blinded landmark oncology trials identified from ASCO, ESMO, and NCCN guidelines. Manuscripts were anonymized by removing identifying information and submitted to each model with a standardized prompt. Two investigators independently analyzed reviews for final verdicts, trial awareness, journal suitability assessments, and concerns across five domains (biological, methodological, statistical, trial groups, and reporting). Chi-square tests with Bonferroni correction and logistic regression were used for statistical analysis. Significant differences were observed in reviewer verdicts among models ( p < 0.001). Claude Sonnet 4.5 was most stringent (74% major revisions, 6% rejections), while Google Gemini 3.0 was most lenient (58% minor revisions, 4% acceptances). Trial awareness varied significantly (Claude 54%, Gemini 50%, ChatGPT 14%; p < 0.001). Methodological concerns differed among models (Claude 86%, ChatGPT 80%, Gemini 62%; p = 0.014). Logistic regression identified statistical concerns as the only significant predictor of critical verdicts (OR = 8.57; 95% CI 1.28–57.4; p = 0.027). Journal suitability was assessed by Claude in 90% of reviews but by Gemini in none. LLMs demonstrate heterogeneous performance in peer-reviewing landmark oncology trials, with significant variation in stringency, trial recognition, and methodological scrutiny. These findings highlight the need for standardized frameworks before integrating AI into peer-review processes.