LLM feedback for academic writing: Effects on students’ performance and engagement

Writing and revising academic texts is a demanding task that benefits significantly from feedback provided by teachers or peers. However, providing elaborated formative feedback on students’ academic writing is time-intensive and therefore hard to implement in educational practice. As a supplementary resource, large language models (LLMs) offer the potential to support the writing process by generating automated feedback to help students enhance their texts. The present study examined the accuracy of LLM-generated feedback on student texts and its effectiveness in improving university students’ revision performance and engagement in academic writing. In a randomized controlled experiment, a sample of N = 144 university students wrote an abstract summarizing a research article. All participants were then instructed to revise their abstracts; half received individualized feedback generated by GPT-4 using a standardized prompting procedure. Controlling for the quality of the initial drafts, regression analyses revealed that LLM-generated feedback led to higher revision quality and increased behavioral engagement, as measured by revision time and edit distance. Furthermore, behavioral engagement partially mediated the effect of feedback on revision quality. These findings demonstrate that LLMs can provide high-accuracy, effective feedback on academic writing. The study discusses the potential applications and implications of this technology within higher education contexts.