Key performance indicators for digital learning software

Digital learning software is used by millions of students worldwide, yet little is known about how consistently students and teachers engage with it beyond initial registration. This, however, is crucial, as continued usage is a prerequisite for learning. Here, we introduce the concept of Key Performance Indicators (KPIs) for evaluating (i) user activity and (ii) usage intensity in digital learning software. We propose a set of KPIs that capture different aspects of user activity and usage intensity leveraging two large datasets ($\approx$13.000 students working through $\approx$22 million exercises; $\approx$2.200 teachers assigning $\approx$44 million exercises) obtained from a classroom-integrated software for learning mathematics in Germany and the Netherlands. We demonstrate how these KPIs serve as measures general enough for monitoring and evaluating any digital learning software with respect to user activity, usage intensity, and implementation fidelity. Importantly, the proposed KPIs also serve accountability purposes by providing insights into whether digital learning software is actually used as intended. In summary, our study underscores the value of KPIs for systematically evaluating the adoption of digital learning software in classrooms or at home. Moreover, we highlight how KPIs provide transparent and comparable evidence, allowing stakeholders, such as policymakers, school leaders, and teachers, to evaluate how digital learning software is used as a prerequisite for learning.