Human versus AI Text Classification from Stylometric Features Across 121,092 Academic Texts

Working paper and open per-text data. This study treats the separation of human-written and AI-rewritten academic text as a plain text classification task. The data are 60,306 human-written academic texts and 60,786 AI rewrites of those texts, produced by eight AI model configurations across six model families, each text described by 47 interpretable stylometric features such as word length, passive voice, lexical diversity, punctuation rates, and sentence openers. A logistic regression on these features separates the two classes with an AUC of 0.936 and an accuracy of 86.4 percent under grouped five-fold cross-validation, and a gradient boosting model increases this to an AUC of 0.961 and an accuracy of 89.5 percent. The errors have a clear structure. Identifying 99 percent of AI rewrites requires falsely flagging 68 percent of human texts, and keeping false flags at 1 percent detects only 47 percent of the rewrites. The misclassified human texts are the most formal ones, and the false positive rate varies almost four times across academic disciplines. Applied without modification to an independent corpus of fully AI generated and hybrid texts, the classifier identifies all 30 generated texts, passes 26 of 30 human texts, and correlates at 0.70 with the mean verdict of nine commercial detectors. Files: per-text classifier scores for all 121,092 texts (opaque pair identifiers; corpus, model, discipline, side, and the out-of-fold scores of both models), the transfer-corpus comparison table, the analysis results, all analysis and figure code, and the paper PDF. The corpus pair texts are not included. Paper page: https://textpulse.ai/research/human-vs-ai-text-classification