Feedback Systems on Computer Programming Courses: A Systematic Literature Review

Feedback is critical throughout the education process. Relevant and timely feedback is difficult to provide in classes with a large number of students, such as computer programming courses. This problem is common in scenarios such as first-year undergraduate education or massive open online courses, where the students have little chance to receive detailed feedback, usually reducing it to a right or wrong answer or to a general discussion about the most common mistakes in the course. Automatically generated feedback can help instructors and learners by personalizing content via just-in-time interventions. Following PRISMA guidelines, this paper used a systematic approach to review the literature about automatically generated feedback for computer programming education. Findings from the study provide a diverse set of approaches for providing feedback, from static ones based on unit tests to dynamic systems using machine learning and Large Language Models. Findings also indicate how different approaches delivered such feedback, including how such systems’ effectiveness was tested (if available), the type of feedback given, and the computational method they implemented. The most common approach was static analysis, with bugs of misconceptions being the most common type of feedback (38%). While not all the reviewed studies tested for educational outcomes, those that did mostly used surveys (>90%) and grades. Such surveys primarily measured participants’ perceptions and motivation. AI-based feedback decision methods are increasingly used, taking advantage of Large Language Models, which enable instructors to provide dynamic feedback. These latest developments will be in permanent improvement as they are personalized, more comprehensive, and provide several types of feedback.