How Multi‑Time‑Slice Experiments Boost Traffic Homogeneity and Reduce Bias

Introduction

As Huolala's experiment complexity grows, multiple time‑slice experiments under the current trading strategy interfere with each other, causing heterogeneous traffic. To prevent these issues and maximize traffic usage, the data‑science team proposes a systematic multi‑layer experiment solution.

1. Time‑Slice Experiments and Capacity Competition

In AB testing, the SUTVA assumption states that treatment of one unit should not affect another. In social AB experiments, user interactions violate this assumption, leading to biased results when, for example, an order‑id split causes the experiment group to capture higher‑quality capacity, inflating its metrics.

Short time‑slices can exacerbate this bias because early slices capture better capacity, leaving later slices with weaker resources. Conversely, overly long slices reduce homogeneity between slices, increasing variance.

2. Variance vs. Bias

When choosing time‑slice length, both capacity competition (bias) and split‑unit homogeneity (variance) must be considered. Short slices increase the number of split units, improving homogeneity (lower variance) but intensifying capacity competition (higher bias). Long slices reduce bias but increase variance due to fewer split units.

3. Experiment Planning Overview

To avoid interference, experiments are physically isolated by city. Some cities run a single experiment, while others run nested experiments. This isolation ensures that experiments in different cities do not affect each other.

4. Management Rules

Front‑end experiments should use ID‑based traffic splitting; if time‑slices are needed, city isolation is required.

New nested experiments must coordinate time‑slice length and group count with existing layers; a maximum of 3‑4 layers is allowed.

City isolation is required when layers are too many, when a single experiment's time‑slice cannot align with others, or when traffic split times differ (e.g., order‑creation vs. driver‑request).

Time‑slice rotation must start at 00:00 to simplify data recovery.

Changes to experiments are only permitted at the end of the current rotation cycle.

Scaling rules: short‑term high‑ROI experiments can be expanded 100% to new city groups at 00:00; lower‑ROI experiments require city‑group swaps after both sides reach a short‑term recovery cycle.

5. Overall Experiment Specification

Experiments must follow the above isolation and nesting guidelines. Nested experiments require aligned time‑slice boundaries across layers, and the upper layer must fully cover the lower layer's slices.

6. Short‑Term and Long‑Term Recovery Cycles

Short‑term recovery considers only the current layer's traffic homogeneity, while long‑term recovery also accounts for interference from other layers, providing a stricter assessment.

Homogeneity is evaluated across dimensions such as order distribution (hour, distance, vehicle type, city tier) and driver availability. Hourly order counts are used as a primary metric, with chi‑square tests applied to assess distribution similarity.

7. Summary

The multi‑time‑slice experiment framework improves traffic homogeneity across experiment groups, shortens observation periods, and balances bias‑variance trade‑offs. By physically isolating cities and carefully planning nested experiments, Huolala can maximize traffic utilization, mitigate inter‑layer interference, and efficiently evaluate experimental impact.