Designing Experiments for Two‑Sided Advertising Markets

Introduction In two‑sided markets, platforms connect supply and demand sides, creating mutual network effects that break the independence assumption of classic A/B tests. The article outlines how Tencent Ads tackles experiment design for such environments.

1. Two‑Sided Market Overview Most internet platforms (ridesharing, e‑commerce, matchmaking, advertising) are two‑sided markets where both supply and demand sides compete and influence each other, making isolation of experimental groups difficult.

2. Four‑Cell Traffic Experiment Traditional geographic, category, or time randomization does not work for ad platforms because ad targeting is not location‑bound, there is no clear category segmentation, and strong temporal spillover (Matthew effect) exists. The four‑cell design splits traffic and ads into groups to observe strategy effects while accounting for capture (抢夺) and spillover (外溢).

3. Solution Approaches To avoid interference, experiments can duplicate ads and isolate traffic, but this incurs revenue loss and engineering complexity. Alternative methods include splitting traffic and ads into multiple buckets (e.g., 50% traffic with 50% ads) or using a three‑part split (experiment, control, blank) to reduce capture while limiting spillover.

4. Counterfactual Interleaving Experiment Based on Facebook’s framework, this within‑subject design interleaves rankings from experimental and control strategies for the same request. While it reduces the need for independent groups, it suffers from Condorcet paradox, loss of high‑value ads, and state‑dependency contamination, especially in feedback‑loop heavy ad systems.

5. Joint Sampling via Contingency Table Extends the four‑cell design to an m×n layout, sampling traffic and ads in multiple proportions to estimate capture and spillover effects. Linear models fit parameters for strategy impact (α), spillover (β′), capture (γ′), synergy (β), and competition (γ), providing a richer view of experiment dynamics.

6. Two‑Sided Market Simulation System Tencent built a lightweight simulation that abstracts the full ad stack (recall, ranking, model prediction, feedback loop) to evaluate experimental designs offline. By running millions of requests under different strategies, the system quantifies expected gains and validates methodological soundness before production rollout.

Overall, the article demonstrates how careful experimental design—considering interference, spillover, and competition—can yield reliable insights for ad strategy optimization in complex two‑sided markets.