How Variance Reduction Boosts A/B Test Sensitivity Without More Samples

Introduction

When running A/B experiments, practitioners often face non‑significant results or excessive metric variance. While increasing sample size is the common remedy, many situations prevent scaling up, prompting the need for variance‑reduction methods.

Why Reduce Variance?

Experiment sensitivity—the ability to detect true treatment effects—improves as the variance of the estimator decreases. Lower variance brings the estimated effect closer to the true effect, making experiments easier to declare significant.

Variance‑Reduction Methods

Techniques are divided into at‑assignment (before allocating users to groups) and post‑assignment (after allocation).

At‑Assignment: Stratified Sampling

Stratified sampling splits the population into non‑overlapping layers and draws proportional samples from each layer, reducing between‑layer variance. The variance of stratified sampling is always less than or equal to that of simple random sampling.

Post‑Assignment: Post‑Stratification

Post‑stratification applies weighting to the sample means after random assignment, using known population layer proportions. The weighted mean formula and its variance are shown below.

Post‑Assignment: CUPED

CUPED (Controlled Experiments Using Pre‑Experiment Data) adjusts the primary metric with a pre‑experiment covariate X, preserving unbiasedness while reducing variance. When the regression coefficient θ equals the optimal slope, the variance of the difference is minimized.

Method Comparison

At‑assignment methods are harder to implement and may not always outperform post‑assignment techniques. Post‑assignment methods (post‑stratification and CUPED) are easier to apply and achieve comparable variance reduction.

Case Studies

Real‑world experiments from companies such as Microsoft Bing, Huolala, and Netflix illustrate how variance reduction can halve the required sample size or accelerate significance detection, especially in “old‑user” scenarios where pre‑experiment data is available.

Summary

Variance‑reduction techniques enable higher experiment sensitivity without increasing sample size, relying on historical data to adjust metrics. They are categorized into at‑assignment (stratified sampling) and post‑assignment (post‑stratification, CUPED) methods, with post‑assignment approaches generally preferred in practice.