Why A/B Tests Fail in Recommendation Systems and How to Fix Them

Complexity of Recommendation A/B Tests

In short‑video platforms, recommendation A/B tests are more intricate than typical experiments because they simultaneously affect multiple roles—consumers, creators, and the platform itself—creating double‑sided and network effects.

Common Pitfalls of Simple Designs

Standard user‑side A/B tests often suffer from learning effects, novelty decay, and metric convergence issues, leading to short‑term gains that do not reflect long‑term performance.

Typical Sources of Bias

Learning effect : Users adapt to repeated recommendations, changing their click behavior over time.

Metric convergence : Long‑term metrics like DAU require extended observation periods, causing early experiments to appear insignificant.

Self‑evolution of the recommendation system : Changes in one ranking stage affect others, making experimental observations differ from full‑rollout outcomes.

Double‑sided effect : Strategies that boost consumer metrics also influence creator behavior, so a consumer‑only experiment misses supply‑side impact.

Network effect : Users in the same social graph share content, contaminating control groups and violating the SUTVA assumption.

Experimental Solutions

Double‑Sided Experiment Design

Separate experiments for users and creators, ensuring that traffic gains for boosted creators come from within the experimental group rather than from control‑group creators.

Three practical ideas are discussed:

Balance total traffic between experimental and control authors.

Run simultaneous user‑side and author‑side experiments to monitor both sides.

Use a Counterfactual Framework to keep both traffic quantity and quality balanced by maintaining two parallel ranking systems.

Community Randomized Experiments (Cluster Randomized Experiments)

To mitigate network effects, users are grouped into communities (clusters) and the entire community is assigned to either treatment or control, reducing cross‑group influence.

Key steps:

Define the business problem and the social interaction to isolate.

Construct a social graph that captures relevant interactions.

Partition the graph into dense, well‑separated communities using the Leiden algorithm.

Randomly assign whole communities to treatment or control.

Validate the uniformity of cluster sizes and numbers (chi‑square, rank‑sum, t‑test).

Evaluate strategy impact using community‑level metrics.

Leiden Community Detection

The Leiden algorithm optimizes modularity (Q) to find high‑quality partitions in massive graphs. It iteratively moves nodes, aggregates them into super‑nodes, and repeats until convergence.

Evaluation metrics include Normalized Mutual Information (NMI), Adjusted Rand Index (ARI) when ground truth is known, and modularity, conductance, and coverage when it is not.

Practical Findings

Community randomization significantly reduces bias from network effects, though it increases metric variance due to smaller effective sample sizes. Double‑sided designs capture supply‑side impacts that consumer‑only tests miss.