The role of statistical power in context: implications for regulatory practices

Regulatory risk assessment in ecotoxicology relies heavily on statistical endpoints such as no observed effect concentrations (NOECs) and effective concentrations (ECxs). Current practices, as outlined in the OECD Guidance Document No. 54 (2006) and related guidelines, typically use decision flowcharts to guide statistical analysis. The flowcharts for NOEC calculations use pretests for normality and variance homogeneity to select appropriate statistical tests from predefined options, including Williams, Dunnett, step-down Jonckheere-Terpstra, and other tests. Although these flowcharts aim to ensure appropriate test selection, our analysis reveals critical limitations in their application to real ecotoxicological data, particularly when dealing with heterogeneous variances and small sample sizes. Through simulation studies and practical data examples, we demonstrate that commonly recommended approaches, including nonparametric tests, can produce misleading results under realistic conditions. Our findings show that robust methods, particularly Welch-type Dunnett's tests, consistently outperform standard approaches when handling heterogeneous variances. Importantly, we emphasize that the goal of statistical analysis should not be to maximize test power for specific data characteristics, but rather to ensure reliable inference across diverse experimental conditions. We propose practical improvements to current guidelines, including (1) prioritizing robust statistical methods that perform reliably under various conditions, (2) incorporating visual inspections and comprehensive sanity checks, and (3) establishing a public repository for benchmark data and methods. These recommendations aim to enhance the reliability and transparency of statistical analyses in regulatory ecotoxicology, ultimately improving the quality of environmental risk assessments.