The Interplay Between Design and Analysis of Experiments in Complex Environments: Interference and Randomization Tests

Experimental design and analysis are two inseparable pillars of causal inference, yet their interplay becomes particularly critical in complex environments where standard assumptions-such as the Stable Unit Treatment Value Assumption (SUTVA)-fail. One such challenge arises from interference, where the treatment assigned to one unit influences the outcomes of others, complicating both the experimental setup and the estimation of causal effects. This article explores how interference impacts both design and analysis, highlighting why both perspectives should be considered to ensure valid inference and improve sample efficiency. On the design side, interference necessitates structured experiments like cluster randomization to account for spillovers. On the analysis side, interference violates independence assumptions, leading to biased estimators and incorrect hypothesis tests, particularly when testing for spillover effects. To address these challenges, randomization-based tests provide a robust, model-agnostic approach by leveraging the experiment's own randomization scheme to construct valid finite-sample inference, featured in the seminar talk by Panos Toulis for understanding the spillover effect of tax audit policies. Recent advances, including graph-theoretic and machine-learningassisted randomization tests, have extended their applicability to more general settings with arbitrary interference structures. These developments are particularly relevant for large-scale A/B testing in online marketplaces and social networks, where interference is prevalent. By integrating thoughtful design with robust analysis, researchers can extract more credible and actionable causal insights from experiments in complex environments.