A large-scale empirical investigation of measurement invariance decisions under multiple-group item response theory and multiple-group confirmatory factor analysis

Measurement invariance (MI) testing is essential for ensuring valid cross-group comparisons in international large-scale assessments (ILSA). There are two major frameworks for establishing MI: multiple-group confirmatory factor analysis (MGCFA) and multiple-group item response theory (MGIRT). This study compares the results of MGCFA and MGIRT in examining measurement invariance using survey data from the International Computer and Information Literacy Study (ICILS) 2023. This study goes beyond prior simulation-based comparisons by providing a large-scale empirical examination of measurement invariance decisions across 33 ICILS questionnaire scales and 32 educational systems under operational assessment conditions. Using common thresholds for model evaluation, the results from the MGIRT suggest invariance across most scales compared to MGCFA, which rejects configural invariance for several scales. This finding suggests that the choice of method, MGCFA or MGIRT, can lead to substantially different conclusions. These findings underscore the urgent need for further methodological research to better understand the conditions under which each approach performs reliably and to guide researchers in making informed choices when assessing measurement invariance.