Peer Review Report For: Can ChatGPT write better scientific titles? A comparative evaluation of human-written and AI-generated titles [version 1; peer review: 2 approved with reservations]

Background Large language models (LLMs) such as GPT-4 are increasingly used in scientific writing, yet little is known about how AI-generated scientific titles are perceived by researchers in terms of quality. Objective To compare the perceived accuracy, appeal, and overall preference for AI-generated versus human-written scientific titles. Methods We conducted a blinded comparative study with 21 researchers from diverse academic backgrounds. A random sample of 50 original titles was selected from 10 high-impact general internal medicine journals. For each title, an alternative version was generated using GPT-4.0. Each rater evaluated 50 pairs of titles, each pair consisting of one original and one AI-generated version, without knowing the source of the titles or the purpose of the study. For each pair, raters independently assessed both titles on perceived accuracy and appeal, and indicated their overall preference. We analyzed accuracy/appeal using Wilcoxon signed-rank tests and negative binomial models, preferences using McNemar’s test and mixed-effects logistic regression, and inter-rater agreement with Gwet’s AC. Results AI-generated titles received significantly higher ratings for both perceived accuracy (mean=7.9 vs. 6.7, p-value