Bootstrap Methods for Statistical Inference in AB Testing

The article introduces the Bootstrap resampling technique as a practical solution for statistical inference in AB testing, especially when dealing with small sample sizes, complex metric constructions, or heavily skewed data.

Common challenges in AB experiments include insufficient sample size, metrics that are ratios of random variables (e.g., CTR = CLICK/PV), and severe data skewness. Traditional hypothesis‑testing methods can address these issues but often become cumbersome.

Bootstrap, a non‑parametric resampling method, solves these problems by repeatedly drawing samples with replacement from the original dataset and computing the statistic of interest.

Core steps of non‑parametric Bootstrap:

(1) Assume the original sample size is N and draw N observations with replacement.

(2) Compute the target statistic T for the resampled data.

(3) Repeat the process B times (typically B > 1000) to obtain B estimates of T.

(4) Summarize the B estimates (e.g., mean, variance) to approximate the original statistic’s distribution.

Bootstrap enables the construction of confidence intervals (CIs) and hypothesis tests for a wide range of statistics.

Three common Bootstrap CI methods:

Standard Bootstrap (SB)

Uses the empirical mean and variance of the Bootstrap replicates to form a CI based on the Central Limit Theorem.

Percentile Bootstrap (PB)

Directly uses the percentiles of the Bootstrap distribution of the statistic to define the CI.

t‑Percentile Bootstrap (PTB)

Combines SB and PB by constructing a t‑type statistic for each Bootstrap sample and using its percentiles, offering improved accuracy and convergence.

A comparison table shows that, for a small‑sample production experiment, all three Bootstrap CIs are narrower than the conventional method, with PTB providing the smallest width.

Application example: In an AB test of a new product feature affecting CTR, 10,000 users were sampled for both treatment and control groups. After 1,000 Bootstrap resamples (each of size 20,000), the distribution of the estimated CTR difference was used to compute a 95% CI via the PB method. The CI included zero, indicating no statistically significant uplift.

The article also contrasts Bootstrap with the Jackknife method, highlighting differences in sampling (with‑ vs. without‑replacement) and suitability for smooth versus non‑smooth statistics.

Conclusion: Although Bootstrap originated decades ago, advances in computing power have revived its adoption. It is now a key non‑parametric tool for AB testing, helping to handle small samples, reduce variance, and simplify inference, with further potential when combined with techniques like DID or Bayesian estimation.