Introduction to A / B Testing

Among all the forms of experimentation, one stands out for its simplicity and power: A/B testing. This chapter unpacks the science behind that simplicity. The strength of any A/B test depends on how fair the comparison is between the two versions. Fairness starts with consistency. Achieving fairness takes intentional design. In most modern experimentation systems, triggering is implemented through what is called counterfactual logging. In practice, most experimentation systems rely on one of three patterns: exposure-based, action-based, or hybrid triggering. For an experiment to be useful, it has to be randomized. Randomization ensures that the result reflects the effect of the change itself, not the composition of the users in it. Statistical significance helps teams tell the difference. Power depends on three main factors: the sample size, the variance of the metric, and the minimum detectable effect.