On the Number of Replications in Resampling Tests and Monte Carlo Simulation Studies

We investigate rejection probabilities of statistical tests based on resampling procedures. Our general framework under consideration covers, in particular, bootstrap and permutation techniques. It turns out that specific properties of the p-value distribution play a key role, namely convexity or concavity, the Bernstein property and those of beta mixture models. We provide a detailed analysis and clarify how these properties relate to each other. We derive new bounds for the rejection probability. The results link the number of replications with size and power of the test. Numerical considerations demonstrate the quality of the bounds. An important application is the nested simulation estimator in Monte Carlo simulation studies. Our findings indicate that a moderate or even rather small number of replications is sufficient to obtain useful simulation results. This enables a substantial reduction of the computational effort in Monte Carlo simulation studies.