Educator use of algorithmic advice in decision making: Results from a virtual course recommendation simulation

Algorithms may be better at prediction than humans in a variety of contexts, but they are not perfect. A deeper understanding of the ways in which educators use and trust algorithmic advice within their professional domain is needed. Educators are a particularly unique professional group, in comparison with the other groups studied in the literature, and will likely be the on-the-ground employees expected to integrate algorithmic advice into their pedagogical planning and decision making. This quantitative study used a 2x2 pretest-posttest experimental design to investigate the way that educators used and trusted algorithmic advice for high school science course recommendations. Participants were randomly assigned to one of two purpose conditions in a simulation environment designed using the Judge-Advisor System (JAS) paradigm to assess the influence of algorithmic advice. Results indicated that approximately 30% of participants did not change their decision at any time and that the remaining 70% tended to not change their decision when given the chance. Additionally, a significant interaction was detected for average confidence change with participants who had more years of teaching experience having higher rates of average confidence change in the positive purpose condition.