Disengagement matters: A response-time informed approach to scoring low-stakes assessments

Low-stakes assessments in K-12 education play a crucial role in monitoring student progress, yet their validity is often compromised by disengaged test-taking behaviors, such as rapid guessing and idling. This study introduces a novel application of the Nested Logit Model (NLM) to integrate test-taking engagement, as indicated by response times (RTs), with item responses to improve ability estimation. Using data from a low-stakes reading assessment in the United States (n = 27,556 students in grades 5 to 8), we compared six scoring approaches, including traditional dichotomous scoring, effort-moderated scoring, and nominal scoring that categorized responses based on RT-informed engagement. Our results demonstrated that nominal scoring approaches, particularly those distinguishing rapid guesses and idle responses, yielded superior model fit, increased measurement precision, and provided nuanced insights into examinee behaviors compared to dichotomous scoring methods. Latent class analysis further identified three distinct engagement profiles—effortful responders, rapid guessers, and idle responders—highlighting the need to address both rapid and idle behaviors in modeling. This study emphasizes the value of leveraging RT data to enhance the accuracy of low-stakes assessments while preserving response information. Findings also suggest the NLM framework as a practical and accessible tool for researchers and practitioners seeking to address disengaged behaviors and ensure the reliability of low-stakes assessments.