Design and Implementation of the AB Experiment Platform at Beike Zhaofang

The document introduces the AB experiment platform (Athena) built for Beike Zhaofang to enable fine‑grained product optimization through controlled A/B tests, ensuring each feature iteration is validated with real user conversion data.

It explains the basic A/B testing concept: two variants are shown to randomly split user traffic, and performance metrics such as PV/UV or conversion rates are compared to decide the winning version.

The platform adopts a layered experiment model inspired by Google’s overlapping experiment framework, distinguishing orthogonal experiments (non‑interfering) from mutually exclusive experiments, and using layers, domains, and buckets to manage traffic isolation and sharing.

Traffic bucketing is described: each experiment’s control and treatment groups form buckets within a layer, and a hash of user ID and layer ID determines the bucket, guaranteeing stable and random assignment.

The architecture consists of four main layers: web (UI for configuring experiments and audiences), API (client and server APIs for experiment decisions, using a uniqId passed via load balancer and stored in Redis), data (real‑time Kafka streams to HBase and offline Hive pipelines feeding ClickHouse for metric aggregation), and storage (MySQL for configs, HBase for real‑time logs, Redis for fast lookups, ClickHouse for analytical queries).

Audience targeting supports dimension‑based AB audiences (device, city, URL) and DMP‑derived user‑profile audiences, with white‑list support for testing specific users.

Gray‑release functionality allows dynamic traffic reallocation based on experiment performance, with a Java SDK that periodically fetches configuration, hashes user IDs to bucket assignments, and supports custom parameters per experiment group.

The platform currently serves over 40 concurrent experiments across multiple departments, providing statistical insights for product and algorithm decisions, though it lacks confidence intervals, full support for mutually exclusive layered experiments, and has high coupling in gray‑release implementation.

Future work aims to address these shortcomings, integrate AB testing more tightly into the development workflow, and gather user feedback for continuous improvement.

Map traffics = new HashMap<>(); // key: experiment name (e.g., "exp1") // value: Traffic object containing group‑to‑bucket mappings Map > traffic = new HashMap<>(); // key: group name (e.g., "group1"), value: set of bucket IDs