How Huolala Built a Scalable A/B Testing Platform with Five Allocation Algorithms

Business Background

Huolala, founded in 2013 in the Greater Bay Area, provides intra‑city and inter‑city freight, enterprise logistics, moving, car sales, and post‑sale services. By April 2022 it operated in 352 Chinese cities, with 660,000 active drivers and 8.4 million active users each month.

The company runs more than 200 business scenarios, most of which require A/B experiments to guide product, algorithm, and strategy decisions. To date the platform has supported over 10,000 experiments, with more than 1,000 active experiments per month.

A/B Experiment Overview

The core idea of A/B testing is simple:

By using controlled variables, randomly assign samples to a treatment group and a control group, collect data, and apply statistical methods to determine which group performs better.

Modern internet A/B designs follow Google’s classic paper “Overlapping Experiment Infrastructure: More, Better, Faster Experimentation”, using concepts of “layers” and “domains” to allocate a portion of traffic for product feature validation.

Huolala A/B Experiment Platform Architecture

The platform follows three stages: pre‑experiment, during‑experiment, post‑experiment.

Pre‑experiment: Users design experiments on the configuration console, setting lifecycle, target audience, group allocation, traffic management, etc.

During experiment: The SDK handles traffic splitting, real‑time event logging, and monitoring data reporting. Data are collected via real‑time or batch pipelines.

Post‑experiment: Collected data undergo significance analysis and OLAP processing to generate significance reports and detailed metric dashboards.

Experiment Configuration Console

The console is the UI where product managers and analysts create experiments, define traffic splits, choose allocation methods, set target metrics, and bound the audience. It also provides real‑time status, traffic flow, anomaly alerts, and diagnostic tools. After an experiment ends, results—including confidence intervals and metric breakdowns—are viewed here, completing the experiment loop.

Traffic Reporting SDK

Developers integrate the SDK by requesting a layer ID, constructing request parameters, and invoking the split‑decision API and event‑logging API. The split‑decision response drives business logic, while logged events feed the evaluation pipeline.

Data Recovery Pipeline

The pipeline relies on Huolala’s big‑data infrastructure. Initially, SDK events are sent to Kafka, then ingested by the real‑time computation platform (FeiLiu) into raw tables. A daily T+1 batch job computes metrics into MySQL for display. Real‑time metrics use Flink to write directly to MySQL. With the maturation of the OLAP engine capable of sub‑second joins on billions of rows, all metrics have migrated to real‑time OLAP computation.

Five Traffic Allocation Algorithms

Scientific experiment results depend on proper sample selection and allocation. Random allocation is the most common method, but Huolala adds four more algorithms to accelerate convergence and maintain scientific rigor:

Time‑slice allocation, Time‑slice round‑robin allocation, Ordered time‑slice round‑robin allocation, Spatio‑temporal allocation

Random Allocation / Enhanced Random

Random allocation hashes a split key (e.g., user ID) with Murmur3, takes mod 100, and maps the result to a bucket pre‑filled according to group ratios. To improve uniformity, a consistent‑hash ring is used, locating the first node clockwise after the hash value.

Time‑Slice Allocation

Instead of a user ID, the allocation key is a time slice (e.g., 1 second). All requests arriving within the same slice are placed in the same group, optionally combined with a city ID to compare regional performance.

Time‑Slice Round‑Robin Allocation

Designed for order‑centric scenarios, this method assigns an entire time slice to a single group, preserving order across peak and off‑peak periods. A day is divided into equal slices (e.g., 3600 s). The number of slices per group follows a preset ratio, and a hash of the experiment ID randomises the slice order to keep results consistent across distributed nodes.

Ordered Time‑Slice Round‑Robin Allocation

This variant lets users explicitly define the sequence of slice‑to‑group assignments, possibly rotating through multiple sequences across periods. It gives full control over traffic distribution while still supporting rapid convergence.

Spatio‑Temporal Allocation

Used for intelligent surcharge scenarios, this approach first randomly splits traffic by a spatial key (e.g., origin‑destination pair) and then applies a time‑slice algorithm within each spatial bucket. Effectively a two‑layer experiment—spatial randomisation followed by temporal allocation—provides fine‑grained control over both dimensions.

Conclusion

The described architecture and allocation algorithms form the core of Huolala’s A/B testing platform, which now supports over 5,000 experiments and more than 15,000 experiment versions across all business lines. Future work will focus on improving experiment design usability, enriching metric sets, and adding AI‑driven traffic optimisation such as multi‑armed bandit techniques.