Why Traditional A/B Tests Fail in Two‑Sided Markets—and How to Fix Them

Background

In product growth, A/B testing is the standard method for evaluating a feature (intervention t) by randomly assigning users to a treatment group and a control group, running the experiment for a predefined period, and comparing key metrics such as daily active users (DAU) or click‑through rate (CTR).

Challenges in Two‑Sided Markets

Many internet platforms operate a two‑sided market (e.g., ride‑hailing drivers ↔ passengers, e‑commerce buyers ↔ sellers, media platforms authors ↔ readers). When experiments are launched on both sides simultaneously, the classical assumptions of single‑sided A/B tests no longer hold.

SUTVA Violation

The Stable Unit Treatment Value Assumption (SUTVA) requires that the outcome of each experimental unit depends only on its own treatment. In a two‑sided experiment a consumer who receives treatment t₁ on the demand side may also be exposed to a supply‑side treatment t₂. Consequently the consumer’s outcome is a function of both t₁ and t₂, breaking SUTVA and biasing the estimated effect.

Cross‑Interference

If the two treatments are correlated, users can experience contradictory conditions. For example, t₁ could encourage passengers to comment on a ride, while t₂ disables the comment feature for drivers. Users in the demand‑side control group then see a disabled comment button, whereas users in the demand‑side treatment group are prompted to comment but cannot do so. This interaction distorts the measured lift for both sides.

Spillover and Cannibalization

Interventions on one side can spill over to the other side. A coupon boost for passengers in a specific district may attract drivers from neighboring districts, reducing driver availability there (cannibalization). Because the driver pool is limited, the observed increase in passenger metrics partly comes at the expense of the control‑side supply metrics.

Mitigation Strategies

Small‑World Partitioning

Physically isolate the market into independent “small worlds” where demand and supply interact only within the same partition. Typical implementations include:

Selecting non‑overlapping cities or regions for the experiment.

Restricting content visibility so that users can only see items from authors belonging to the same partition.

Advantages: restores SUTVA within each partition. Caveats: the reduced pool may change baseline metrics (e.g., recommendation quality drops when the author pool shrinks). Practitioners should run a parallel “loss‑measurement” experiment to quantify the impact of partitioning and, if the effect is positive, gradually scale the experiment to larger markets.

Counterfactual Interleaving (Facebook)

Instead of measuring treatment and control separately, interleave the ranking results of both groups into a single list and observe user interactions on that blended list. By comparing the observed click distribution to the expected distribution under no interference, the method estimates the overall lift while accounting for cross‑side effects.

Model‑Based Corrections

Statistical models can be used to estimate and subtract spillover effects. A typical workflow:

Collect pre‑experiment baseline data for both sides (e.g., driver order distribution, buyer‑seller transaction volume).

Fit a regression or hierarchical model that predicts the outcome as a function of the treatment indicator and covariates capturing cross‑side activity (e.g., number of coupons issued, driver density).

Use the model to predict the counterfactual outcome for the control side had the spillover not occurred, then compute the adjusted treatment effect.

Example: after a coupon experiment, compare the spatial distribution of driver orders before and after the intervention; the shift in driver locations can be quantified and used as a correction term.

Conclusion

All three approaches have trade‑offs. Small‑world partitioning offers a clean experimental design but may reduce ecological validity. Counterfactual interleaving leverages existing ranking pipelines but requires careful statistical inference. Model‑based corrections preserve the original experiment layout but depend on the correctness of the underlying model. Practitioners should assess the magnitude of cross‑side interference, weigh implementation cost against expected bias reduction, and select the most appropriate mitigation technique for reliable two‑sided experiment results.