Adaptive Grouping Method for AB Testing in Didi’s Experiment Platform

In data‑driven internet companies, AB testing is a key tool for decision analysis. This article introduces Didi’s AB testing platform and focuses on a new method that improves the uniformity of group allocation during the sampling stage.

The article first outlines the typical AB testing workflow (design, grouping, treatment, analysis) and points out that the grouping step often uses Complete Randomization (CR). While CR is simple, it can produce groups with imbalanced characteristics (e.g., GMV distribution), which harms the validity of downstream analysis.

To mitigate this risk, a simple Re‑randomization (RR) approach is described: repeat CR until the inter‑group imbalance falls below a preset threshold. RR reduces imbalance but incurs extra computation and still cannot guarantee a sufficiently balanced split.

The core contribution is the Adaptive grouping algorithm developed by Didi’s AI Lab. Adaptive grouping performs a single pass over the population while maintaining, for each group, the current sample count and the distribution of selected observation metrics. For each incoming user, the algorithm computes a direct‑allocation score and an indirect‑allocation score for every group, then selects the group with the highest combined probability, ensuring that the metric distributions remain balanced across groups.

The algorithm consists of the following steps:

Shuffle the population and assign the first 2 × K samples (K = number of groups) randomly to guarantee at least two samples per group.

Initialize group parameters: direct allocation probability, indirect allocation probability, and current metric distribution.

For each subsequent sample, calculate a balance score (BS) based on the current proportion of allocated samples. If BS differs greatly among groups, perform a direct allocation using the smallest BS; otherwise, compute a metric‑distribution score (MS) via an ANOVA‑based model and allocate indirectly.

After assigning a sample, update the group’s metric distribution.

The Adaptive method is integrated into Didi’s Apollo AB testing platform. The system receives experiment definitions via API or UI, creates grouping tasks, fetches required metric data from Hive, runs the Adaptive algorithm, and writes the final group assignments to HDFS.

Experimental results show that Adaptive grouping achieves a >95% probability of meeting the balance criterion in a single pass, outperforming CR (up to 14% imbalance) and RR (max 2.7% imbalance). In tests with 10,000 driver samples, Adaptive reduced the 95th‑percentile group‑difference to below 0.8%, compared with 14% for CR and 2.7% for RR.

The article concludes that Adaptive grouping significantly improves the precision of random‑group experiments, reduces the likelihood of invalid experiments, and shortens experiment cycles. Future work includes re‑balancing already‑completed CR experiments.