Unlocking Growth: How AB Testing Validates Causality and Measures Impact

What is AB Testing

AB testing, also known as a controlled experiment, originated from biomedical double‑blind trials and was first applied to Internet product testing by Google engineers in 2000. It has become a core method for data‑driven product growth in companies such as Apple, Amazon, Facebook, Baidu, Alibaba, and many others.

In the online context AB testing (or Online Controlled Experiment) randomly assigns users to different groups, records their behavior via log tags, and compares key metrics to evaluate the effect of a product change.

Types of AB Tests

By product form: app, PC, web page.

By execution mechanism: front‑end page, back‑end service.

By traffic split target: user, session, page, element.

By service invocation: SDK, API service.

By experiment content: interaction, algorithm, content, engineering performance.

Three Fundamental Elements

Experiment participation unit – the users (or other entities) that are randomly divided into a control group and one or more treatment groups. Units must be independent, randomly assigned, and present in sufficient quantity to achieve statistical power.

Experiment control parameters – the variables that are deliberately changed in each treatment (e.g., banner colour, font size). Parameters must be assignable and easily modifiable.

Experiment metrics – the quantitative indicators used to judge the outcome. Metrics must reflect the experimenter’s intent and be measurable with reasonable accuracy.

Two Core Values of AB Testing

Qualitative causality : AB testing isolates the causal effect of a product change, avoiding reliance on intuition or correlation alone.

Quantitative growth : By measuring the magnitude of metric changes, even a 1 % lift can be compounded over time, enabling precise cost‑benefit analysis.

Two Key Characteristics

Prior (pre‑evaluation) : Small‑traffic experiments provide early feedback before a full rollout, reducing risk and saving resources.

Parallelism : Multiple experiments can run simultaneously on the same user base through orthogonal randomisation, increasing efficiency.

Causal Inference Foundations

AB testing relies on the potential‑outcome framework (Rubin Causal Model) and causal‑graph models (Pearl). Under random assignment, the average causal effect (ACE) can be identified by comparing observed outcomes of treatment and control groups.