Digital Assessment Mode Effects in K-12: Insights from Large-Scale NAEP Data in Reading, Mathematics, Science, and Social Sciences

The increasing digitalization of educational assessments necessitates a thorough understanding of mode effects. This study investigates the impact of digital versus paper-based administration using large-scale, randomly controlled data from the U.S. National Assessment of Educational Progress (NAEP). Analyzing data from approximately 930,000 students in grades 4, 8, and 12 across 1385 items in reading, mathematics, science, civics, geography, and U.S. history, we compared performance on digitally based assessments to paper-based assessments. Results indicate that digital administration generally leads to significantly lower student performance across most subjects and grades, with greater impact at lower grades and differential impact across subjects. For instance, the largest mode effect was observed for reading at grade 4, whereas mathematics showed larger effects than other subjects at grades 8 and 12. After accounting for mode effects for the overall population, differential impacts for major student subpopulations (defined by gender, race/ethnicity, disability status, English learner status, and socioeconomic status) were small and mostly non-significant. The mode effect was largely consistent across content areas and items within an assessment. However, students' self-reported perceptions of test difficulty and effort did not consistently align with observed performance differences. These findings highlight that digital assessment can introduce a systematic negative mode effect, underscoring the critical importance of considering such effects in the increasing digitalization of education.