Mastering A/B Testing in Two‑Sided Markets: Principles, Cases, and Strategies

Introduction

In two‑sided market business scenarios, experiments are often needed to evaluate revenue impact and iterate strategies, but the diversity of goals and market complexity makes choosing an allocation method challenging. This article clarifies suitable experiment designs and their theoretical foundations for common company use cases.

AB Test Fundamentals

Understanding relevance versus causality is essential. Relevance describes the degree of association between variables, while causality refers to a direct effect of one variable on another. To establish true causality, confounding variables must be eliminated so that the observed impact can be attributed solely to the target variable.

An AB test isolates a single variable, removing other influences, thereby providing reliable data on the causal effect of the treatment.

Case Study: Game Advertising

A game company wants to increase sales by adding a game‑advertising intervention. The analysis considers:

Treatment : the ad exposure.

Confounder : prior gameplay, which may affect both ad exposure and purchase.

Random assignment of users to see the ad breaks the link between the confounder and the treatment, allowing measurement of the ad’s causal impact.

The experimental model can be expressed as:

outcome1 = confounding + treatment + bias

outcome0 = confounding + bias

Thus, Treatment Effect = outcome1 - outcome0 .

Allocation Principles for AB Experiments

1.1 Experience Consistency

When interventions differ greatly in user experience, inconsistent experiences can cause negative perception or discrimination. Sensitive strategies such as pricing or incentives should ensure fairness and long‑term consistency.

1.2 Novelty Effect

The novelty effect (Von Restorff effect) means users remember and favor new features, leading to short‑term metric spikes that may later converge or reverse. Experiments must account for this effect.

1.3 Homogeneity Between Test and Control Groups

AB tests require comparable groups; otherwise, observed differences may stem from group composition rather than the treatment. Examples include small sample sizes in surcharge experiments or driver‑ID segmentation in ride‑hailing.

1.4 Controllable Experiment Cycle

The experiment duration influences product iteration speed. Some scenarios need multi‑week experiments to obtain statistically sound conclusions.

Overview of Experimental Methods

Different allocation methods align with various principles, each bringing distinct benefits and risks. Selecting the appropriate method depends on the specific business scenario and experimental objectives.