Can Time‑Slice Experiments Skew Your Results? Understanding Capacity Competition and Optimal Design

Introduction

In social AB experiments, interactions between users can violate the SUTVA assumption. Time‑slice (time‑slot) experiments split traffic uniformly over time, reducing interference but may introduce capacity competition.

Capacity Competition Problem

When a time slice allocates high‑quality drivers early, later slices inherit reduced capacity, amplifying differences between groups.

Current experiment uses 10‑minute slices within a 1‑hour rotation.

Order completion time 50th percentile = 53 min, 90th percentile = 93 min.

Key Questions

Principles and steps of homogeneity testing.

Required experiment duration for various slice sizes and rotation methods.

Theoretical Exploration of Capacity Competition

Capacity competition does not affect qualitative conclusions but can distort quantitative results, especially when slice length is short.

Impact depends on the ratio M/N (order completion time M vs slice length N): M/N = 1/3 → 2/3 of later traffic unaffected; M/N = 1/2 → half unaffected; M/N = 1 → all affected.

Variance vs. Bias under Time‑Slice Rotation

Bias arises from shared capacity influencing the control group; variance stems from finer granularity of random splits.

Short slices → lower variance but higher bias.

Long slices → higher variance but lower bias.

Experiment Cycle Evaluation

Two approaches:

Post‑test assessment: Fit historical supply‑demand curves to judge homogeneity.

Pre‑test estimation: Use AA simulation on historical data to predict pairing‑rate convergence.

Findings from AA Simulations

Tested seven slice lengths (10 min, 30 min, 1 h, 2 h, 3 h, 6 h, 24 h). Recommendations:

For slices < 1 h, fixed order + next‑day reversal works best.

For larger slices, fixed order alone introduces bias; reversal improves results.

Slices ≥ 2 h still show significant capacity competition even after many days; avoid > 2 h unless using day‑level experiments.

Practical Guidelines

Design experiments considering slice length, rotation method, and required observation period (typically ≥ 1–2 weeks, with sufficient sample size). Use chi‑square tests for small samples and R² > 0.99 for curve similarity.

Summary

Time‑slice experiments do not change qualitative conclusions but can amplify quantitative gains of advantaged groups. Choosing appropriate slice length and rotation reduces capacity competition and balances variance‑bias trade‑offs.