Spatial Temporal Eye-Tracking Indicators of Novice Programmers’ Skill

As Large Language Models (LLMs) become more common in education, there is growing concern about how to accurately assess learners’ skills, particularly in programming, where understanding cognitive strategies is essential. Traditional measures often overlook the sequential and temporal nature of how students read and comprehend code. This study presents a deep learning-based framework that models spatial-temporal eye-tracking data using long short-term memory (LSTM) and Transformer architectures to classify programming skill levels. We examine both binary (high vs. low) and ternary (high, average, low) labelling strategies to assess their impact on classification performance and instructional relevance. Findings from a stimulus-wise evaluation show that binary classification leads to better accuracy and F1-scores due to clearer separation between classes. LSTMs generally perform better than Transformers across tasks, likely because of their ability to track temporal patterns in smaller, noisier datasets. Although the study uses a single dataset, testing across multiple programming tasks suggests the method can adapt to different problem types. These results support the use of gaze data and sequence models for identifying learner skill levels and informing personalized programming instruction.