Why Write an A/B Experiment Whitepaper? – Overview and Methodology

Growth and optimization are perpetual goals for enterprises, and data‑driven A/B testing has become an essential tool for strategy validation, product iteration, algorithm improvement, and risk control. The whitepaper aims to share the theory, workflow, core elements, and advantages of A/B experiments to promote a culture of experimentation and enable precise decision‑making.

Beyond spreading experimentation culture, the document provides deep insights for researchers facing complex constraints such as small sample sizes, spillover effects, and fairness risks, drawing on Meituan’s fulfillment and delivery business experiences.

The whitepaper covers an overview of A/B testing, fundamental principles, case analyses, and accompanying SDK code, targeting data scientists, system developers, product managers, and anyone interested in data‑driven growth.

Whitepaper Table of Contents

Foreword – Why Write an A/B Experiment Whitepaper?

--- Part I: A/B Experiment Overview ---

1.1 Understanding A/B Experiments
1.2 Deep Dive – Trustworthy Experiments in Meituan Delivery

--- Part II: Fundamental Principles and Case Studies ---

Chapter 2: Basics of A/B Experiments

2.1 Overview of Experimental Principles
2.2 Statistical Foundations of A/B Testing
2.3 Common Terminology

Chapter 3: Randomized Controlled Experiments

3.1 Classic RCTs
3.2 Advanced Techniques to Increase Power
3.3 Methods to Ensure Homogeneity
3.4 Complex RCTs for Spillover Effects
3.5 Extensions and Outlook

Chapter 4: Randomized Rotation Experiments

4.1 Coin‑Flip Rotation
4.2 Full Random Rotation
4.3 Paired Rotation
4.4 Extensions and Outlook

Chapter 5: Quasi‑Experiments

5.1 Difference‑in‑Differences
5.2 Extensions and Outlook

Chapter 6: Observational Studies

6.1 Synthetic Control
6.2 Matching Methods
6.3 Causal Impact
6.4 Extensions and Outlook

Chapter 7: Advanced Experiment Tools

7.1 Meta‑Analysis
7.2 Multiple Comparisons
7.3 Extensions and Outlook

--- Part III: SDK Code Application ---

Chapter 8: Open Analysis Engine

8.1 Product Features
8.2 System Architecture
8.3 System Integration
8.4 Offline Analysis Case Study

1.1 Understanding A/B Experiments – A/B testing (Online Controlled Experiment) originates from double‑blind clinical trials and has been widely adopted since Google’s 2000 rollout. It involves randomly assigning users to control and treatment groups to isolate the causal impact of a new feature.

1.2 Deep Dive – Meituan Delivery Example discusses common pitfalls such as small samples, spillover effects, variance mis‑calculations, and fairness risks. It explains how randomization, proper variance estimation, and ROI‑oriented metrics are essential for reliable conclusions.

The document also describes a zero‑threshold, trustworthy experiment workflow that enables data scientists, data‑warehouse engineers, and system developers to collaboratively design, configure, and monitor experiments, providing automated significance testing, trend analysis, and diagnostic tools.

Finally, a method‑selection guide recommends prioritizing randomized experiments, followed by quasi‑experiments and observational studies, with a visual flowchart and a summary table of applicable methods, many of which are integrated into the fulfillment SDK.