Online Traffic Splitting AB Testing: Design, Implementation, Evaluation, and Decision

Background: AB testing compares two variants under identical conditions, with online traffic splitting (online) and offline sampling (offline). This article focuses on the online traffic‑splitting framework.

1. Experiment Design

Define experiment purpose, decide metrics (invariant vs. variable), design metric aggregation, and allocate traffic. Discuss metric types: conversion, per‑user, aggregate, and ratio. Explain traffic acquisition via random split and sample‑size estimation, including formulas for minimum sample size.

2. Experiment Implementation

Choose random unit (page, session, user) based on user experience and metric importance. Describe orthogonal versus mutually exclusive experiments and provide rules with diagrams.

Conduct robustness checks using AA tests and overlapping hierarchical bucketing to ensure unbiased buckets.

3. Experiment Evaluation

Statistical description covers data quality, cleaning, handling missing values, outliers, noise, and data transformation (normalization, discretization, sparsification). Basic descriptive statistics (mean, median, mode, range, variance, skewness, kurtosis) and frequency, trend analyses are outlined.

Statistical inference includes hypothesis testing, p‑value interpretation, statistical power (1‑β), confidence intervals, and methods for deeper analysis such as dimension segmentation and attribution.

4. Experiment Decision

Based on core metric significance, dimensional analysis, and business considerations, decide whether to roll out, scale, or iterate the experiment.

Conclusion and community invitation.