Using webcam eye tracking and AI to determine how students proceed in basic mathematical activities: a study based on a digital adaptive learning system

Abstract Many students struggle with basic mathematics, such as the understanding of numbers and arithmetic operations, even in grade 5 and higher. Previous research has shown that the difficulties are often reflected in how the students proceed when working on basic mathematics tasks. Eye tracking (ET) has proven valuable for analyzing how students proceed in such activities, for example, number line activities. However, dedicated ET equipment is typically not accessible for teachers and analyzing ET videos is too time-consuming and demanding for in-service educators. The aim of this paper is to investigate how a combination of webcam-ET and AI can help; specifically, we aim to investigate how well AI (here: Supervised Machine Learning, SML) can determine from webcam-ET data how students proceed. We developed a digital learning system called KI-ALF that uses webcam-based ET and AI to automatically identify how students proceed and collected data from 141 fifth graders, which were analyzed for this study. Our findings show that the AI is promising in determining how students proceed, yet, the quality of the identification varies between different mathematical activities, with the highest performance for number line activities (and lower performances for enumeration in structured whole-number representations and visual multiplication activities). This indicates the subdomain-specificity of the opportunities of AI to automatically identify the ways students proceed in basic mathematics activities. The paper presents the results in detail and identifies task-specific factors that contribute to the opportunities that AI offers in this respect.