Optimal Weighted Tests for Replication Studies and the ‘Two‐Trials Rule’ With Multiple Hypotheses

ABSTRACT Replication studies for scientific research are an important part of ensuring the reliability and integrity of experimental findings. In the context of clinical trials, the concept of replication has been formalized by the “two‐trials” rule, where two pivotal studies are required to show positive results before a drug can be approved. In experiments testing multiple hypotheses simultaneously, control of the overall familywise error rate (FWER) is additionally required in many contexts. The well‐known Bonferroni procedure controls the FWER, and a natural extension is to introduce weights into this procedure to reflect the a‐priori importance of hypotheses or to maximize some measure of the overall power of the experiment. In this article, we consider analyzing a replication study using an optimal weighted Bonferroni procedure, with the weights based on the results of the original study that is being replicated and the optimality criterion being to maximize the disjunctive power of the trial (the power to reject at least one non‐null hypothesis). We show that using the proposed procedure can lead to a substantial increase in the probability of success (PoS), is robust to changes in the effect sizes between the two studies, and recovers the submission wise type I error rate of the two‐trials rule.