Assessing inconsistent responding in digital and paper formats: a replication study using the TIMSS 2023 Confidence in Mathematics scale in Ireland

This study replicates and extends previous research on inconsistent responding in mixed-worded scales. Using a posttest-only experimental design, we examine the prevalence of inconsistent responding on the TIMSS 2023 Confidence in Mathematics Scale by comparing randomly equivalent digital and paper questionnaire administrations in Ireland. Two detection methods are employed: Mean Absolute Difference and Factor Mixture Analysis. Results indicate slightly higher rates of inconsistent responding in digital than in paper administration formats across most comparisons. Also, fourth-grade students exhibited higher rates of inconsistent responding compared to eighth-grade students. The study further investigates the role of student characteristics in predicting inconsistent responding. Among student background and digital-specific variables considered in binary logistic regression models, mathematics achievement was the only systematic negative predictor of inconsistent responding, explaining most of the variance, especially in younger students. This study highlights the importance of considering administration mode in large-scale assessments and provides implications for survey methodology, especially as digital testing becomes more widespread.