Using Bayesian Knowledge Tracing to Evaluate Perceptual Scaffolding and Desirable Difficulties in a Math Intervention

The present study examines the assistance dilemma of providing combinations of perceptual supports in math education by fitting a Bayesian knowledge tracing model (BKT) to an open-source, online order-of-operations intervention dataset. This online intervention dataset contained 688 U.S. middle schoolers' accuracy data who solved order-of-operations problems in pretest, intervention, post-test, and delayed post-test sessions. Combinations of spacing and color cues were embedded in intervention materials, alongside immediate correctness feedback and worked examples, creating nine between-subject cue conditions (congruent, incongruent, or neutral spacing and/or color). To examine the effect of perceptual cues on immediate performance and long term learning, we trained a BKT variant, conditioning guess, slip, learn, and forget parameters on perceptual cue conditions. The estimated cue-specific BKT parameters suggest two main findings. First, the presence of congruent perceptual cues (color and/or spacing, e.g., 2 - 8 + 3×11) increased guessing rate and reduced slipping rate, suggesting that visually highlighting high-order suboperations facilitated students' immediate problem solving. Second, students who underwent intervention with incongruent color cues (e.g., 2 - 8 + 3 × 11) showed reduced guessing rate but exhibited the highest learning rate and the lowest forgetting rate. These results imply that providing misleading visual highlights may hinder immediate performance but can be beneficial for long term learning. Furthermore, the impact of perceptual cues on immediate performance and long term learning indicated by BKT parameters approximately aligned with the patterns revealed by behavioral measures (i.e., intervention, post-test, and delayed post-test accuracy). Such consistencies demonstrate the feasibility of using the BKT framework to understand the assistance dilemma of perceptual supports. The code and data are available at: osf.io/q8kn6.