Building a One‑Stop AB Testing Platform at NetEase Cloud Music: Architecture, Metric Infrastructure, Scientific Evaluation, and Efficiency

NetEase Cloud Music created a one‑stop AB testing platform to meet the fast‑iteration demands of its product teams, replacing an earlier, limited system. The platform integrates experiment management, data collection, and result analysis into a unified workflow.

Platform Infrastructure : A high‑performance traffic splitter supports online split‑testing across iOS, Android, wearables, car‑head units, and TVs. An automated data‑recovery pipeline aggregates experiment data, avoiding siloed metric definitions, and leverages the Doris analysis engine and ClickHouse for end‑to‑end analytics.

Metric Infrastructure : Metrics are categorized into atomic (e.g., impressions, clicks, plays) and composite (e.g., click‑through rate, average play time). A drag‑and‑drop development interface on the batch‑stream processing platform enables rapid metric creation, and all metric data are stored in ClickHouse for ad‑hoc querying.

Scientific Evaluation : The platform addresses key scientific challenges such as sample uniformity (SRM), correct metric calculation, and robust effect assessment. It implements uniform traffic allocation, monitors SRM via chi‑square tests on dimensions like age and device, and provides alerts. Statistical power is improved through larger sample sizes, variance reduction techniques (CUPED, outlier removal), and multiple‑testing corrections (Bonferroni, BH).

Experiment Efficiency : Efficiency is enhanced by improving integration speed, collaboration boundaries, and decision‑making processes, fostering a strong experimentation culture.

QA Section : Answers common questions about shared control groups and metric ownership, emphasizing BI‑driven metric definitions with product and operations input.

The presentation concludes with acknowledgments and promotional material for the 2023 Data Intelligence Innovation & Practice Conference.