CONSTRUCTIVE BY DESIGN? EVALUATING AN AI FEEDBACK COACH IN COLLABORATIVE LEARNING CONTEXTS

Read full text Summarise with VIRAT
Authors: Danny J. Scholten, Sonja van Scheijen, Mary-Jo Diepeveen, Fanny Beekman, Andréa E. Tijhuis, Jacqueline E. van Muijlwijk‐Koezen
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Given that LLMs (large language models) can produce fluent, scalable, and context-sensitive responses on request, often simulating human conversation, many higher education institutions are considering their use for feedback purposes. It is tempting to consider LLMs as suitable replacements or supplements for limited teacher feedback in large student cohorts. However, LLM-based educational technologies are being introduced faster than their consequences can be fully characterized; effects on student-agent interactions, feedback quality, and learning outcomes remain insufficiently understood. As such, we investigated an LLM-based AFC (Automated Feedback Coach) that provides feedback-on-feedback while students compose peer-review comments in an undergraduate academic skills course. In a mixed-methods case study, we compared an intervention cohort using the AFC with a control cohort without it. Students’ and lecturers’ perceptions and reported feedback behavior were examined through a student survey and focus groups, supplemented by lecturer notes. To assess effects on student work, peer feedback comments were analyzed using quantitative language and sentiment metrics, along with qualitative coding of clarity, constructiveness, and tone. Results show that students largely accepted the AFC, describing it as useful and trustworthy and reporting gains in confidence and perceived feedback quality. In contrast, analyses of peer-feedback comments showed only minor differences: the AFC cohort produced slightly more complex feedback, without meaningful improvements in clarity, constructiveness, or tone. Nevertheless, qualitative evidence indicated negative effects on the learning process, as the AFC seemed to elicit assurance-seeking and tool-satisficing behavior rather than critical reflection. Moreover, students and lecturers perceived the AFC as distracting during the peer feedback writing process and reported variable AFC feedback quality, including generic or inconsistent guidance and sensitivity to differences in phrasing. Based on our findings, we recommend a cautious, evidence-informed approach to developing and integrating LLM-based agents into formative assessment strategies, rather than adopting them solely on the basis of prevailing trends or student evaluations.

Fields of Study & Subjects

Student Assessment and Feedback Artificial Intelligence in Education Intelligent Tutoring Systems and Adaptive Learning

Publication details

Journal / source
INTED proceedings
Volume
1
Publication date
1 March 2026
Language
EN