The validity of school socio-economic status measures: how do indicators from TIMSS and PISA compare to national administrative data sources?

Abstract Background Policymakers and researchers collect data on student and school socio-economic background for various purposes, such as prioritising students from socio-economically disadvantaged backgrounds for additional resources, understanding the background characteristics associated with educational achievement, or monitoring the progress of policy reforms and initiatives. International large-scale assessments gather detailed information from participating students and school principals and use this to calculate composite measures of socio-economic status. At the same time, policymakers usually have access to national socio-economic indicators based on administrative data. Given the impact of socio-economic indicators on monitoring and shaping education policy, key issues related to the convergent and criterion-related validity of the international and national socio-economic measures are of concern to both policymakers and researchers. Methods In this paper, we compare socio-economic indicators from the 2019 Trends in International Mathematics and Science Study and the 2022 Programme for International Student Assessment with two key indicators used in policy-making in Ireland (the Pobal HP Deprivation Index and school participation in the Delivering Equality of Opportunity in Schools programme) and examine their associations. Furthermore, we consider the extent to which the indicators explain variance in mathematics achievement. Analyses include correlations, linear and logistic regression, and multilevel modelling. Results Findings show strong associations between the socio-economic indicators examined, with stronger associations among indicators derived from aggregated student-level data ( r ≈ 0.7) than among inherently school-level indicators ( r ≈ 0.6). Principals’ judgements about the proportion of disadvantaged students in their schools, collected via TIMSS and PISA, are found to be quite accurate insofar as these estimates align well with both national indicators and other ILSA-derived aggregated indicators. International indicators were more strongly predictive of achievement in mathematics than the national indicators, with the national indicators explaining little additional variance in TIMSS and PISA mathematics outcomes beyond the ILSA-derived SES indicators. Conclusions This paper provides valuable empirical validity evidence for both international, and Ireland’s national, socio-economic indicators. The methodological approach presented here may also be valuable as a model for other countries participating in international large-scale assessments, particularly those that use their own administrative socio-economic data for policy-related decision-making.