Reporting Practices, Open Science Practices, and Trustworthiness of Simulation Studies in Psychology: A Questionnaire Study

Monte Carlo simulation studies in which statistical methods are evaluated, are widespread in scientific literature. Surprisingly, they have received remarkably little attention from a meta-research perspective. Those studies that did investigate the current state of replicability, reproducibility, and reporting standards in simulation studies often argue for reform of standard practices to improve their quality (e.g., Luijken et al., 2024; Morris et al., 2019; Pawel et al., 2024). To date, however, no study has examined the prevalence and motivations for selective reporting and uptake of open science practices in simulation studies, specifically among researchers who have themselves conducted simulation studies. Moreover, no previous literature has investigated the trust researchers have in simulation studies conducted by others in their field. This study sets out to increase our understanding in these areas. This exploratory study investigates the prevalence and motivations for selective reporting, adherence to open science practices, and reasons for non-adherence in simulation studies in psychology that evaluate statistical methods. We also examine the level of trust in simulation studies conducted by others in the field, among researchers who have themselves conducted at least one such study. This preregistration serves as a guide for both the authors and readers in conducting and evaluating the analyses. While we aim to be as transparent as possible in outlining our analysis decisions and protocols prior to data collection, the exploratory nature of this study implies that additional analyses will be conducted next to the preregistered analyses. Any additions to the preregistered analyses will be documented in the final publication.