Offline Sampling in AB Testing: Challenges and Experimental Techniques

In the context of AB testing, “offline sampling” refers to determining the assignment of experimental and control groups before the experiment starts. Unlike online traffic‑based experiments, offline sampling deals with pre‑defined units such as users, ad plans, or tags, making the grouping stable throughout the test.

Why offline sampling is needed? It is commonly used when the intervention’s effect spans multiple user visits, such as product changes that must be consistent for each user, or operational activities like sending red packets where the allocation (receive or not) is decided in advance.

Main challenges of offline sampling

1. Insufficient sample size : The number of sampled units (e.g., users) is often an order of magnitude smaller than the number of page views in online streaming, leading to low statistical power.

2. Heterogeneity of sampled units : Large variance among units (e.g., merchants vs. consumers) can cause imbalance between treatment and control groups, especially when a few high‑value units dominate key metrics.

3. Non‑random intervention allocation : Business rules or user behavior may dictate who receives the treatment, introducing confounding factors that violate randomization assumptions.

Experimental techniques to address these challenges

3.1 Variance reduction : Increasing sample size is the simplest way, but techniques like CUPED (Controlled‑experiment Using Pre‑Experiment Data) leverage pre‑experiment covariates to reduce the variance of the treatment effect estimator.

When the sampling distributions of the metric under treatment and control heavily overlap, the test power is low. Reducing variance (e.g., via CUPED) can separate the distributions and increase power.

3.2 Stratified sampling and Inverse Probability Weighting (IPW) : By dividing the population into homogeneous strata (e.g., based on historical ad spend) and sampling within each stratum, heterogeneity is mitigated. IPW adjusts the weights of units in each stratum during metric calculation to correct for imbalance.

3.3 Matching methods : Direct matching pairs each treated unit with a control unit that is similar across all covariates, effectively controlling for confounders. When exact matching is infeasible, propensity‑score matching provides a single summary score to facilitate matching.

These methods together help overcome the three major difficulties of offline sampling, though the maturity of offline techniques still lags behind online traffic‑based experiments.

Conclusion

Offline sampling is essential for many internet business analyses where the experimental unit is defined before the test. The main obstacles are limited sample size, strong heterogeneity, and non‑random treatment assignment. Variance‑reduction techniques, stratified sampling, IPW, and matching (including propensity‑score matching) are practical tools to improve statistical power and causal inference in such settings.