This article explains how pre‑experiment bias can distort AB test results and introduces a suite of techniques—including AA retrospective analysis, SeedFinder optimal random grouping, variance reduction, and an offline splitting algorithm—to create homogeneous test groups and improve experiment reliability.
This article explains why variance reduction is essential for A/B experiments, describes at‑assignment and post‑assignment techniques such as stratified sampling, post‑stratification and CUPED, compares their effectiveness, and presents real‑world case studies demonstrating how they improve experiment sensitivity without increasing sample size.
This article explains the statistical foundations of A/B testing, introduces potential outcomes and average treatment effect, defines metric covariance, and presents practical estimation methods—including naive, data‑augmentation, and bucket‑based approaches—along with real‑world performance evaluations and applications such as variance reduction and Bayesian optimization.
This article explains how Meituan's fulfillment team designs, implements, and operates a reliable A/B testing platform for multi‑side markets, detailing statistical pitfalls, experiment types, traffic-splitting frameworks, and automated analysis pipelines to ensure credible results despite overflow effects, small samples, and fairness constraints.
The Sustainable Online Reinforcement Learning (SORL) framework tackles offline inconsistency in auto‑bidding by iteratively gathering safe online data from real ad systems with a Lipschitz‑based exploration method and training a variance‑suppressed conservative Q‑learning policy, achieving safer, more stable, and higher‑performing bids on Alibaba’s platform.
The article describes Tubi’s internal experimentation platform, explaining how traditional user‑group A/B tests can suffer from network interference and how Switchback experiments—time‑window based designs—address these issues, detailing their implementation, statistical methods, and the practical challenges overcome.
This article explains how the law of large numbers and the central limit theorem underpin Monte Carlo methods, revealing why their convergence rate is low, how significance and confidence levels are defined, and why variance reduction is crucial for efficient simulations.
This article explains how adding covariates that are not confounders can reduce outcome variance and improve causal inference, while controlling for variables that only predict treatment may introduce selection bias and inflate estimation error.
The article explains offline sampling for AB testing, detailing why it is needed, the main challenges of limited sample size, population heterogeneity, and non‑random interventions, and presents variance‑reduction, stratified sampling, IPW, and matching methods to address these issues.
Offline sampling in A/B testing assigns experimental units such as users or tags before a trial begins, but suffers from limited sample size, high heterogeneity, and non‑random allocation, which can be mitigated by variance‑reduction methods like CUPED, stratified sampling with inverse‑probability weighting, and matching approaches including propensity‑score matching.
