Non-Native English Writing and the False Positives of Stylometric AI Text Detection

Read full text
Authors: TextPulse Research
Cited by: 15
References: 9

Abstract

Working paper and open per-essay data. A widely cited study found that perplexity-based AI text detectors flag the majority of essays by non-native English writers as AI-generated while sparing native writers. This study measures whether the bias is true for a transparent stylometric classifier. We score all 5,600 essays of the ICNALE Written Essays corpus, written under controlled conditions by college students from ten Asian countries and regions at known proficiency levels and by English native speakers, with the human-versus-AI classifier of our prior studies, trained on 50,701 academic texts and rewrites and never on any essay. Every essay predates modern AI writing tools, so every flag is a false positive. The logistic regression classifier shows no bias against non-native writers. It flags 2.7 percent of learner essays and 3.3 percent of native essays at the default threshold, statistically indistinguishable rates, and both far below the 10.3 percent it flags on full-length human academic texts and the 24.3 percent on length-matched academic excerpts. The proficiency gradient runs opposite to the published bias, as the most proficient learners are flagged the most, namely 5.0 percent at B2 against 1.9 percent at A2, since proficient writing drifts toward the formal register the classifier associates with an AI writing style. Learner limitations, namely simple everyday vocabulary, repetitive word choice, and uneven sentence lengths, are the opposite of the AI signature under a stylometric analysis, while a perplexity analysis reads the same limitations as machine-like predictability. The protection is not intrinsic to the feature set, however. A gradient boosting model trained on identical features, texts, and folds flags learners at 12.5 percent against 3.8 percent for natives, a threefold gap with the least proficient flagged most. Professional native-speaker editing of learner essays raises the linear model's flag rate from 2.1 to 4.0 percent while lowering the boosted model's. Whether AI detection discriminates against non-native writers is therefore a property of the detector itself, not of the human writing style. The published bias is explicit for the AI detectors it was measured on, but it is a consequence of specific detector designs rather than a general law of AI detection. Version 1.1 is a wording and clarity pass over the full text. No data, code, figures, or numerical results changed. The full paper is published at https://textpulse.ai/research/ai-detection-non-native-writers. This record holds the paper PDF, the per-essay classifier scores for all 5,600 ICNALE essays, the 656 edited pairs and the length-matched academic controls (corpus identifiers and scores only, no essay text), the per-group metrics, and all extraction, analysis, and figure code. The ICNALE corpus itself is distributed by its authors under terms that prohibit redistribution of the texts and is not included here.

Fields of Study & Subjects

Authorship Attribution and Profiling Text Readability and Simplification Artificial Intelligence in Healthcare and Education

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
22 August 2026