Design and Implementation of an AB Testing Platform with Traffic Allocation Algorithms

Background : In product design and business decision making, choices such as button color or placement need data‑driven validation. Traditional voting or leader decisions are not reliable; AB testing provides a scientific alternative.

Business Value – R&D Perspective :

Prior knowledge : Small‑traffic tests validate ideas before full rollout, reducing product loss.

Parallelism : Multiple versions can be tested simultaneously under identical environments, shortening verification cycles.

Scientific rigor : Statistical metrics decide experiment success, avoiding reliance on intuition.

Business Value – PM Perspective :

Systematic experiment workflow lowers the barrier for data‑driven product decisions.

Visualization and automated interpretation of results reduce manual analysis effort.

Core Concepts :

Scene – independent business scenarios (e.g., homepage recommendation).

Bucket – a container within a scene; multiple buckets can coexist, traffic between buckets is mutually exclusive.

Layer – a set of experiments sharing the same traffic pool; layers are orthogonal to each other.

Experiment – a subset of traffic used to evaluate a specific feature or strategy.

Traffic – all user requests.

Orthogonal traffic – traffic allocation across layers does not interfere (MECE).

Experiment Modeling : User traffic is divided into three parts – hierarchical bucket, small‑traffic bucket, and baseline bucket. Experiments sit at the intersection of a layer and a bucket, each layer uses a salted hash to ensure stable user‑experiment mapping. Baseline experiments handle unmatched traffic.

Traffic Allocation Algorithm : Users are hashed (user‑id + layer‑specific salt) and the modulo result (0‑100) determines the bucket interval. Simple fixed intervals work well for static traffic, but adjusting percentages can cause “pollution” (e.g., reducing A’s share shifts users from B to C). The improved algorithm follows two steps:

Step 1 – Prefer own interval : Each experiment first claims traffic within its original interval according to the new percentage.

Step 2 – Fill gaps : Remaining unfilled traffic is allocated to neighboring gaps, minimizing impact on other experiments.

This “multi‑withdraw‑fill” approach keeps most users in their original experiments, reducing metric distortion during successive adjustments.

System Design :

All decision logic runs in local memory; only a configuration center and a database are external dependencies.

Configuration reload flow: edit experiment → DB update → static config version bump → push to config center → all machines reload cache.

Design principles: weak consistency (few‑second delay acceptable), small cache size, read‑heavy/write‑light workload.

Potential issues: database spikes when many machines reload simultaneously, push latency growth with more subscribers, and occasional “zombie nodes” that miss config pushes.

Mitigations include random staggered reloads, proactive cache loading, selective query of recent changes, and monitoring of node health.