Design of Full-Traffic AB Experiments for Seller Growth on Xianyu

Background : Xianyu, China’s largest second‑hand marketplace, relies heavily on sellers for supply. Evaluating strategies that boost seller growth requires scientific AB testing, but traditional designs struggle with shallow‑inventory, supply‑side experiments.

Business background and traditional AB : Sellers differ from buyers; their needs are urgent, they determine platform supply, and seller transactions yield higher retention. Conventional AB splits traffic by buyer ID, which cannot ensure a seller appears in only one bucket, leading to biased results.

Principles for persuasive AB experiments :

The same seller/item must not experience different strategies across buckets.

Supply allocation must match traffic allocation.

Intervention should not harm the control group.

Full‑traffic AB design : Assign sellers to two groups by hashing seller ID (50% each). All buyer buckets run identical code containing an if‑else that applies the experimental logic per seller. This keeps supply and traffic split consistent and isolates the intervention to the experimental sellers only.

Case study – churned seller tilt : New or churned sellers who received a “I want it” chat incentive showed a significant retention lift. A full‑traffic AB compared sellers A1 (experiment) vs B1 (control) on chat UV, transaction UV, and retention.

Results over five days: chat‑UV share increased by +22% for the experimental sellers; overall per‑user buyer‑seller metrics rose by ~1%, indicating no adverse impact on overall transaction efficiency.

Further discussion : When experiment logic is scattered across many services, a lightweight single‑bucket AB can be used at the cost of some theoretical guarantees. Alignment of seller‑side metrics with buyer‑side metrics may diverge under full‑traffic AB, but comparisons remain valid within the seller groups.