Experimental Design for Two-Sided Markets in Advertising Scenarios

This article explores experimental design challenges in two-sided markets, particularly in advertising scenarios. A two-sided market connects two groups (supply and demand sides) where their behaviors influence each other through network effects, making traditional A/B testing problematic due to independence assumptions.

The article presents several experimental approaches. The four-table experiment divides traffic and ads into groups, but faces issues with competition and spillover effects. Counterfactual interleaving, inspired by Facebook's framework, runs two algorithms in parallel but suffers from Condorcet paradox and state dependency problems.

The proposed solution is contingency table joint sampling, which generalizes the four-table approach by using m×n designs with upper triangular sampling. This method estimates competition and spillover effects, supports both ad-level and traffic-level experiments, and can assess strategy impacts on the entire ecosystem.

The article also introduces a two-sided market simulation system developed by Tencent, which abstracts and simplifies the advertising system to test experimental designs before implementation. This simulation can evaluate different experimental methods and verify their effectiveness in complex market scenarios.

Key challenges in two-sided market experiments include engineering complexity, sample size limitations, and the difficulty of achieving true independence between experimental and control groups. The article emphasizes the importance of proper experimental design for making reliable decisions in advertising platforms.