Reporting Practices, Open Science Practices, and Trustworthiness of Simulation Studies in Psychology: A Questionnaire Study

Simulation studies have received little attention in the replication crisis. This exploratory study investigated reporting practices, open science practices, and trustworthiness of simulation studies in psychology. 221 researchers (response rate=35.1%) completed a questionnaire about one simulation-based article they authored in 2023-2024. Authors reported at least some selective or fragmented reporting of conditions, methods, and performance measures in 50.2% of the articles (i.e., omitting results completely or split between the main text and supplements). Reporting nonconvergence and Monte Carlo Standard Errors was uncommon, likely due to limited awareness and weakly established norms. 19% of articles were neutral (i.e., authors did not develop the evaluated methods), but selective and fragmented reporting was not lower in these studies. Over 75% of authors reported sharing their complete code, but measures for reproducibility like providing instructions (39.6%) and independent code checking (20.3%) were uncommon. Guidelines were not used because researchers were unaware of them, perceived them as unnecessary, or considered them uncommon. These prevalence estimates should, however, be interpreted with caution, as self-report and self-selection bias may have led to overly optimistic estimates. Respondents estimated the probability that conclusions of a typical simulation study are trustworthy to be .74, exceeding estimates for computational reproducibility (.64) and complete reporting (.55). As responsible reporting standards and code-sharing were not standard, we recommend journals to endorse reporting standards such as ADEMP (Morris et al., 2019), use standardized repositories and workflows for sharing materials, and check code availability.Keywords: simulation studies, reproducibility, replicability, open science, QRPs, selective reporting, trustworthiness