Reporting AI in education research: a methodological audit of 2025-2026 publications against an adapted TRIPOD-LLM checklist

Read full text
Authors: Iryna S. Mintii, Dmytro V. Verbovetskyi, Ostap Yu. Sirenko
Cited by: 1
References: 16

Abstract

We audit how the use of artificial intelligence is reported in recent education research. From a harvest of 29 848 arXiv preprints and 543 articles in six education and education-technology journals (2025-2026), we coded 220 papers (127 arXiv + 93 journal) against a 19-item checklist adapted from the TRIPOD-LLM reporting guideline, plus descriptive and outcome items. Coding was performed by open-weight large language models (served through Ollama) from titles and abstracts, conservatively (an item not stated is coded 0); we report cross-model agreement (mean Cohen's κ = 0.53, raw agreement 87%) in place of inter-human reliability, and disclose the AI-coded method in full. Overall reporting compliance is low: the median paper reports 32% of the checklist items, and the lowest-compliance items are the cross-cutting accountability signals - funding and conflicts of interest, missing-data handling, calibration/fairness, compute and cost, and the human-in-the-loop protocol (each ≤ 7%). Reporting quantity does not differ between arXiv preprints and journal articles in the unadjusted comparison (equal medians; unadjusted odds ratio ≈ 1); what differs is composition - preprints document the model machinery while journal articles document the study context, and neither documents accountability. A modest journal advantage in quantity emerges only after adjusting for study design. Empirical design is the dominant predictor of how many items a paper reports. A within-paper preprint-vs-published comparison was planned but could not be conducted, as no eligible pairs exist. We contribute the TRIPOD-LLM-for-education checklist - to our knowledge the first reporting checklist derived from TRIPOD-LLM and calibrated for general (non-medical) education research - as a citable artefact, and call on education journals to require accountability reporting at submission.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Academic Publishing and Open Access Academic integrity and plagiarism

Publication details

Journal / source
CTE Workshop Proceedings
Publisher
Academy of Cognitive and Natural Sciences
Volume
13
Pages
236–255
Publication date
21 March 2026
Language
EN