Do AI Detectors Agree? An Inter-Rater Reliability Study of Commercial AI Text Detectors on Academic Writing

Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid human-AI texts agreement falls to chance level (mean pairwise kappa 0.02). In total, 28 of 30 hybrid texts received both a human verdict and an AI verdict from different tools, and the median per-text score range was 100 points. On mixed human-AI writing, the classification a text receives relies more on the choice of detector than on the text itself. This record contains the working paper (PDF), the full 90-text corpus with per-text provenance, the 90 x 9 detector score matrix, the generation protocol, and the deterministic analysis and figure scripts. Canonical page: textpulse.ai/research/ai-detector-agreement Version 1.1 adds nine screenshots of corpus hybrid text SS1 scored in each detector's own web interface (account details blurred), matching the case study shown on the paper page, and updates the paper PDF to the final published byline. Version 1.2 retakes the Originality.ai screenshot on the same hybrid text (SS1) as the other eight detectors.