A Controlled Comparison of AI Text Humanizers on Academic Writing

Working paper and open data. AI text humanizers rewrite machine-generated text so that AI detectors classify it as human. Humanizer vendors advertise pass rates, but no published study measures what the rewriting does to the output text itself. We passed 48 AI-generated academic texts (about 300 words each, four disciplines, three citation styles, four AI model generator families) through eleven humanizers under one setting per tool, and scored 432 outputs on a fixed set of metrics, namely, position on a stylometric human-to-AI spectrum; semantic faithfulness to the source; retention of technical terms, numbers and citations; fabricated numbers and named entities; Flesch-Kincaid grade shift; grammar errors introduced; length change; rewrite depth; hidden-character injection; and vocabulary shift against a 1,057-word AI vocabulary lexicon generated from our prior study. Detection was measured with Turnitin and GPTZero, the gold standard university-grade detectors. Every humanizer tool pushed the text toward the human end of the spectrum, reduced AI-leaning vocabulary and lowered the reading grade, and the tools differed most in what they preserved. TextPulse rewrote 52 percent of the words while keeping 97 percent of the document meaning, 99 percent of citations and 87 percent of numbers, introduced 0.65 grammar errors per 1,000 words, fabricated almost no named entities, and removed more AI-leaning vocabulary than any tool that kept the text intact. The tools with the highest raw GPTZero pass rates had the weakest content: Walter Writes passed 60 percent of its outputs but also fabricated 4.4 numbers or names per each (~300 word) text, and Phrasly passed 38 percent but introduced 3.6 grammar errors per 1,000 words. On a strict experiment, when a detector pass is counted only for outputs that also preserve meaning, citations and facts, TextPulse has the highest rate at 10.4 percent (95 percent CI 2.1 to 18.8), Phrasly and GPTinf have 2.1 percent, and eight tools have none. TextPulse is also first on an equal-weight composite of rewrite depth, faithfulness, spectrum position, grammar cleanliness and content integrity, with or without the detector component. On Turnitin, a document holding only TextPulse's outputs scored 24 percent AI, the lowest score in the field together with StealthWriter; the other tools scored from 28 to 74 percent. Detector pass rate on its own is an incomplete measure of a humanizer, since the easiest approach to pass is to change what the text says, or even intentionally inject grammar errors. Across all experiments, all of the humanizer tools had the same treatment to allow for an even level playing field. Every metric was scripted, and all humanization outputs and code are made publicly available for easy replication of the same experiments in future work. Files: the paper (PDF), per-text scores for all 432 humanized outputs, the analysis results, a data archive with the 48 source texts, verified ground truth, every humanized output and every GPTZero reply, and the scripts and figures.