Text Length and the Reliability of Human versus AI Text Classification in Academic Writing

Working paper and open per-text data. Our previous study classified human-written and AI-rewritten academic texts from interpretable stylometric features with an AUC of 0.936 on texts with a median length near 290 words. This study measures how that reliability depends on text length. From 25,561 text pairs in which both the human text and its AI rewrite contain at least 300 words, we cut every text to 25, 50, 100, 150, 200, 250, and 300 words, re-extract 45 stylometric features at each length, and retrain the classifier at each length under grouped five-fold cross-validation. The composition of the sample is identical at every length, so the results isolate text length from every other property of the texts. Classification reliability increases monotonically with length. The AUC of a logistic regression increases from 0.817 at 25 words to 0.940 at 300 words and 0.950 on the untruncated full texts, and most of the increase is at the 150-word length. Reliable individual verdicts degrade faster than the AUC suggests. Identifying 95 percent of AI rewrites requires falsely flagging 34 percent of human texts at 300 words, but 68 percent at 25 words, and a strict one percent false positive budget identifies only 17 percent of AI rewrites at 25 words. A classifier trained on full-length texts and applied to short texts at a fixed threshold flags 70 percent of 25-word human texts as AI while its detection of AI rewrites hardly changes, since short texts of any authorship have mechanically high lexical diversity, which the classifier reads as an AI characteristic. Word-level features preserve most of their distinguishing power at short lengths, and sentence-level variation carries almost no signal under 50 words, where texts average fewer than three sentences. The full paper is published at https://textpulse.ai/research/ai-detection-text-length. This record holds the paper PDF, the per-text out-of-fold classifier scores at every length band for the matched and mismatched designs, the per-band metrics, and all analysis and figure code.