Design and Architecture of an Overlapping Experiment Platform for Data‑Driven Product Operations

Background

With rapid business growth, product operations at Hujiang increasingly rely on data‑driven, fine‑grained methods such as A/B testing to achieve stable user growth, order conversion, and revenue.

Design Idea

The platform is built on two core experiment elements: traffic (user visits) and parameters (experiment variables). To run many experiments simultaneously without interference, a layered model inspired by Google’s Overlapping Experiment Infrastructure is adopted.

More: scalable to run many experiments with flexible traffic allocation.

Better: quickly detect and stop ineffective experiments.

Faster: enable non‑engineers to create experiments, accelerate result reporting, and support rapid traffic scaling.

The model divides traffic into domains and layers: a non‑overlapping domain with a single layer and an overlapping domain with three layers, allowing up to three concurrent experiments per request.

Key concepts:

Domain – a traffic partition.

Layer – a subset of system parameters.

Experiment – a combination of parameters that a user will encounter.

In the overlapping domain, each request can belong to at most one experiment per layer, and traffic is re‑shuffled at each layer.

Experiment Platform Architecture

The platform consists of four modules: experiment management, traffic management, data collection (reporting to a Hadoop cluster), and data analysis (including confidence interval and P‑value calculations).

Workflow:

Product or developers configure experiments and traffic allocation via the AB console; configurations are stored in the database.

The AB sharding service loads configurations into memory and refreshes periodically.

Business services request sharding decisions from the AB service.

Clients report event data to the BI system.

The reporting platform reads BI data, performs statistical tests, and generates reports.

Experiment Metrics

Common metrics such as order conversion, retention (1‑day, 3‑day, 7‑day), and click‑through rates are supported and can be configured to produce daily reports automatically.

Scientific Validation

Statistical significance is assessed using P‑value tests and confidence intervals. The article explains hypothesis testing, the role of the significance level (α = 0.05), and how Z‑tests (or T‑tests for small samples) are derived from observed data.

Confidence intervals indicate whether an experiment’s effect is statistically significant and in which direction.

Summary

A/B testing compares different product solutions and requires clear design differences.

Define core metrics for tracking.

Implement comprehensive instrumentation for reliable data collection and analysis.

References

Google paper on overlapping experiments.

Online A/B confidence interval calculators.

Resources on confidence intervals and P‑value testing.