Development of a tool to assess the risk of bias in statistical simulation studies

Abstract Background: Statistical simulation studies are the principal methodology for examining the performance of statistical methods. Findings from statistical simulation studies guide researchers in their statistical decision making, and inform evidence syntheses of statistical simulation studies, so should provide a fair representation of how the methods are expected to perform. Objectives: To develop a tool to assess whether statistical simulation studies provide a fair representation of how statistical methods are expected to perform. Methods: We will undertake a multi-step process to develop a domain-based tool with signalling questions. The project team will consist of a core working group and an international advisory group. We will hold virtual meetings with the advisory group to agree on the scope and content of the tool. We will conduct systematic reviews and cited reference searches (forward and backward citations) to develop an evidence base to inform domains and signalling questions. An initial set of domains will be agreed on with the advisory group. We will then undertake a survey of methodologists and statisticians with expertise in statistical simulation design and intended users of the tool, to seek their views on which domains are most important. We will propose signalling questions for each domain and revise the domains with feedback from the advisory group until domains are agreed. We will pilot the tool with intended users such as consulting statisticians and systematic reviewers of findings from simulation studies. The developed tool and guidance documentation will be published in an open-access journal and disseminated via conferences and workshops.