Detecting ChatGPT in Scientific Text: A Historical Perspective on Chatbot Catchphrases

Large language model (LLM)-generated text, beyond grammar and spelling improvements, may be present in scientificdocuments even without attribution. Hence, development of an accurate manual screening paradigm would be helpfulto detect whether LLM, or chatbot, content is included. In this study, examples are provided to suggest how peculiarcatchphrases detected in scientific text can be checked for chatbot generation. Catchphrases were first selectedmanually, and then Google Search was utilized in seeking articles with each catchphrase up to 2024 to show a historicalperspective. Thereafter, it was determined whether paragraphs with the catchphrase were of chatbot origin using theGPTZero detector. The number of articles published with each catchphrase in recent literature, citations per article, thepublishing journal Impact Factor, and the document section in which the chatbot phrase appeared were compiled tocharacterize chatbot phrasing. In this investigation, it was found that most suspected peculiar phrasings were indeedchatbot-associated. Based on a statistical analysis, the number of published biomedical and bioengineering articles withchatbot phrasings has increased substantially in recent years, particularly after the onset of ChatGPT. Moreover, mostof the chatbot-containing articles studied were published in journals with a substantial Impact Factor. Chatbot-generatedtext is most commonly found in the Abstracts and Introduction sections, but also in the Methods, Results andDiscussion, Limitations, and Conclusions. It is therefore inferred that Chatbot content has peculiar phrasing that can bedetected manually, and that it ubiquitously appears in scientific documents in the peer-reviewed literature. Chatbotgeneratedcontent, over and beyond simple grammar and spell correction, is present and has been increasing even intop-ranked journals.