Investigating the analytical robustness of the social and behavioural sciences

Read full text Summarise with VIRAT
Authors: Balazs Aczel, Barnabas Szaszi, Harry T. Clelland, Marton Kovacs, Felix Holzmeister, Don van Ravenzwaaij, Hannah Schulz-Kümpel, Sabine Hoffmann, Gustav Nilsonne, Livia Kosa, Zoltan A. Torma, Yousuf Abdelfatah
+88 more Christopher L. Aberson, Oguz Ali Acar, Ensar Acem, Matus Adamkovic, Timofey Adamovich, Krisna Adiasto, Love Ahnström, Atakan M. Akıl, Adil S. Al-Busaidi, Ali H. Al‐Hoorie, Casper Albers, Peter Allen, Taym Alsalti, Micah Altman, Shilaan Alzahawi, Ettore Ambrosini, Saule Anafinova, Rahul Anand, Martin Angerer, Ariadna Angulo-Brunet, Alberto Antonietti, József Arató, Andreu Arenas, Marco Mendoza Aviña, Flávio Azevedo, Marko Bachl, Bence Bagó, Štěpán Bahník, Bradley James Baker, Elza Balayan, Cassandra Baldwin, Benjamin Banai, Kasia Banas, Frantis̆ek Bartos̆, Ernest Baskin, Jojanneke A. Bastiaansen, Nadège Bault, Christopher W. Bauman, Quintin Beazer, Maciej Behnke, Theiss Bendixen, Sebastian Berger, Zoé Bernard Anna Juliette, Ursa Bernardic, Paul Alexander Bloom, Annika Boldt, Ciril Bosch-Rosa, Rotem Botvinik‐Nezer, Adam Zaid Austin Bouyamourn, Özge Bozkurt, Laurel Ellen Brehm, Johannes Breuer, Ryan C Briggs, Hilmar Brohmer, Erin Michelle Buchanan, Johannes Buckenmaier, Jeffrey Buckley, Jacek Buczny, Matthias Burghart, Bilal Hayat Butt, Nick Byrd, Valentina Cafarelli, Patrick M. Callahan, Tabaré Capitán, Kevin R Carriere, Andrea Michael Cataldo, Gabriel Cepaluni, Eugene Y. Chan, Jesse Chandler, Chia-chen Chang, Xi Chen, Shirley (Shuo) Chen, Fadong Chen, Hao Chen, Valerii Chirkov, Daniela Cialfi, Beth Clarke, Sophie G. Coelho, Clara Cohen, Jason Collins, Susan Cook, Gaia Corlazzoli, Jamie Cummins, Christian S. Czymara, Jonathan D'hondt, Anna Dalla Rosa, Abi M. B. Davis, Fruzsina Venczel
Cited by: 0
References: 43

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1–3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Fields of Study & Subjects

Meta-analysis and systematic reviews Psychology Research and Bibliometrics Statistical Methods and Bayesian Inference

Publication details

Journal / source
MetArXiv (OSF Preprints)
Publication date
31 March 2026
Language
EN
Document type
Preprint