Supplementary material from "Code sharing and reproducibility in survey-based social research: evidence from a large-scale audit"

Reproducibility—the ability to obtain original results by reapplying the original analyses to the original data—is an essential component of empirical research. In this study, we assess the reproducibility of articles using the European Social Survey (ESS), a large-scale repeated cross-sectional dataset widely used across the social sciences. Drawing on more than 1,000 ESS-based articles published between 2015 and 2020, we investigate whether authors share their code for reproduction purposes and whether published results are reproducible. We find that only about one in three authors (35%) share code. Of 699 empirical results from 100 studies with code, only about half (51%) are numerically reproducible, while the others are either dif-ferent (26%) or technically irreproducible (23%). Overall, about one in six published results is exactly re-producible. Numerical deviations are usually minor and do not indicate systematic bias. Reproducibility is similar across disciplines, with pronounced reproducibility problems also in disciplines that have received little prior monitoring. Reproducibility failure mostly stems from unavailable, poorly documented, or in-complete code. We propose low-cost measures for authors, editors, journals, and data providers to improve code availability and reproducibility in large-N observational social research.