Ensuring Homogeneity in AB Tests: Practical Solutions for Reliable Results

Introduction

In many AB experiments at Huolala, small experimental units and coarse traffic splitting cause large pre‑experiment bias (also called “homogeneity”), which can lead to misleading strategy evaluations.

1.1 Definition of Homogeneity

Homogeneity refers to the degree of difference between treatment and control groups’ observed metrics before any intervention. If the difference is not statistically significant, the experiment is considered homogeneous; otherwise, it is heterogeneous.

1.2 Homogeneity Testing Methods

Different metrics require different significance tests. For ratio, continuous, proportion, or median metrics, appropriate tests such as chi‑square, t‑test, or non‑parametric tests are used. The choice of test is summarized in the following table.

1.3 Practical Workflow

The workflow consists of three steps:

Select historical experiment objects that have exhibited the relevant behavior.

Partition these objects into multiple groups so that historical metric differences between groups are minimal.

Apply a homogeneity‑ensuring technique (offline AA backtracking, optimal grouping, etc.) to generate the final grouping or random seed.

2.1 Pre‑Experiment Homogeneity Assurance

Identify the most relevant metric and back‑track period, then ensure the historical metric differences between groups are sufficiently small. Offline AA backtracking or optimal grouping can reduce pre‑experiment bias. Choose core metrics first and be aware of multiple‑testing issues.

Determine the back‑track period by splitting historical data into two parts: the first part (length m) is used to find the period that maximizes correlation with the second part (length n, equal to the expected experiment duration). Linear regression R² is used to evaluate fit, and the optimal m is selected as the back‑track window.

2.2 Post‑Experiment Homogeneity Assurance

When core metrics become heterogeneous after launch, techniques such as CUPED or DID can quantitatively assess and correct homogeneity. For side‑effect metrics, outlier removal is recommended to eliminate experimental noise.

Side‑effect metric homogeneity: test for significant differences using real‑experiment data; no historical data needed.

Case: PK experiment – supply‑demand homogeneity tested with chi‑square on hourly execution volume and driver push count.

2.3 Experiment Cases

Case study: “Cash‑on‑Delivery Guarantee” experiment – users selecting cash‑on‑delivery must pay a proportion of the order amount. The experiment design, data flow, and homogeneity checks are illustrated.

Summary

For homogeneity assurance, use offline AA backtracking or optimal grouping before the experiment to reduce pre‑experiment bias. After launch, apply CUPED/DID for core metric heterogeneity and outlier removal for side‑effect metric heterogeneity. These practices improve the reliability of AB test results and support sound decision‑making.