A Comprehensive Guide to A/B Testing: System Design, Implementation, and Best Practices

Introduction In the data‑driven era, product iterations and strategic decisions require quantitative evaluation, and A/B testing provides a scientific method to compare new and existing versions.

A/B Test Overview

A/B testing originates from double‑blind clinical trials, where subjects are randomly split into control and treatment groups to assess significant differences. In internet products, multiple design variants are served to randomly assigned user groups, and their behavior and business metrics are collected for analysis.

AB System Design and Implementation

2.1 System Introduction

The core functions of the ZuanZuan AB Test system include:

Experiment Management: configuration, launch, and offline operations.

Metric Management: creation and management of event and composite metrics.

Whitelist Management: creation and management of whitelist identifiers.

Data Reporting: overview of experiment users, traffic, key metric charts, and conclusions.

Split Service: RPC service for business side to obtain experiment group results.

2.2 System Architecture

2.3 System Implementation

2.3.1 Experiment Management

Experiment List : consists of filter/query area, new experiment area, and list area.

New Experiment : requires basic info, configuration info, and strategy config; default status is "testing".

2.3.2 Metric Management

Metrics are divided into "event metrics" (collected via event tracking) and "composite metrics" (derived from arithmetic operations on event metrics).

2.3.3 Whitelist Management

The whitelist feature provides unified creation and management of identifiers used to include users in specific experiment groups, simplifying testing.

2.3.4 Data Reporting

Experiment reports contain basic information (ID, name, start time, run days, operation logs, configuration view) and core data (total users, group users, traffic allocation, core metric values, confidence interval, statistical power, and conclusions).

Basic Information

Core Data

2.3.5 Split Service

Split Logic

if (白名单判断) {
    return "白名单组";
}
if (实验下线判断) {
    return "决策组";
}
if (进组不出组) {
    if (缓存结果判断) {
        return "缓存结果组";
    }
}
// 分桶分组，用实验id + 分流标识进行hash取模100得到桶号，同一用户在不同实验中的桶号不完全一样，确保实验之间的独立性
int bucketNum = BucketNumUtil.getBucketNum(testId + "_" + tokenId);
// 根据桶号获取对应的实验组
String groupName = getGroupName(test, bucketNum);
if (进组不出组) {
    redisCache.set(testId, tokenId, groupName, exAt);
}
return groupName;

Split Scheme

ZuanZuan uses a "no‑layer" scheme where each experiment occupies a full set of buckets, shuffled with the experiment ID as seed.

// 生成1-100的桶号，并使用testId作为种子洗牌打乱
List
list = Stream.iterate(1, item -> item + 1).limit(100).collect(Collectors.toList());
Random rnd = new Random(testId);
Collections.shuffle(list, rnd);
// 根据组流量比例将桶号分配到各组
for (int i = 0; i < groups.size(); i++) {
    // TODO 按照流量占比分配相同数量的桶号
}

To handle mutually exclusive experiments, a "mutual‑exclusion group" concept will be introduced, with new logic shown below.

if (白名单判断) {
    return "白名单组";
}
if (实验下线判断) {
    return "决策组";
}
int groupBucketNum = BucketNumUtil.getBucketNum(groupId + "_" + tokenId);
if (!互斥组流量判断(groupInfo, groupBucketNum)) {
    // 不在互斥组流量中实验时，返回对照组
    return "对照组";
}
// ... same bucket calculation as before ...
return groupName;

A/B Test Implementation Guide

Each step of an experiment is critical; the overall workflow is illustrated in the diagram below.

3.1 Experiment Design

3.1.1 Purpose

For internet products, each new version launch must be evaluated scientifically using data and statistical principles to decide which version performs better.

3.1.2 Design Template

3.1.3 Structure

The experiment design consists of four parts: basic information, configuration information, metric information, and strategy design.

Basic Information : business line, experiment name, ID, objective, etc.

Configuration Information : experiment type, expected launch time, duration, and metric definitions (core, related, guardrail).

Metric Information : core metrics (directly tied to goals), related metrics, and guardrail metrics (one‑vote veto).

Strategy Design : group naming, version description, traffic allocation (up to 100%, 1% granularity).

3.2 Experiment Event Reporting Specification

Example JSON payload for event reporting:

{
    "埋点事件名":"ab",
    "埋点内容":{
        "实验页面":"XXXX",
        "实验id":"XXXX",
        "实验分组":"XXXC",
        "分流用户类型":"XXXX"
    }
}

3.3 Experiment Decision Guide

3.3.1 Decision Process

After an experiment runs, its overall effect is evaluated using the experiment report, which includes traffic distribution, core and related metric statistical tests, confidence intervals, and statistical power.

3.3.2 Confidence Interval & Statistical Power

Confidence Interval provides a range for the true metric difference; a 95% confidence level (Z=1.96) is commonly used.

Confidence Level

Z Value

99%

2.58

95%

1.96

90%

1.645

80%

1.28

If both bounds are positive, the result is significantly positive; if both are negative, significantly negative; if the bounds cross zero, the difference is not significant.

3.4 Experiment Expiration and Offline

Two scenarios:

Decision Offline before expiration : when statistical power is high, the winning group is deployed directly.

Automatic Offline after expiration : if power is low, the experiment defaults to the control group.

Future Plans and Outlook

Experiment Type Diversification : more experiment types for different business scenarios.

Richer Data Reports : add retention, funnel, and attribution analyses.

Real‑time Data and Alerting : move from T+1 offline data to near‑real‑time monitoring.

Conclusion

This article covered what A/B testing is, the design and implementation of an AB platform, practical guidelines for running experiments, and future development directions. In a competitive product landscape, A/B testing enables low‑cost, data‑driven decisions to attract new users, retain existing ones, and iterate quickly.

Author: Wang Zhiyuan, Senior Data R&D Engineer at ZuanZuan, responsible for big‑data platform construction, event‑tracking standards, and real‑time data development.