Evaluating Long-Term vs Short-Term Effects in A/B Experiments

With the widespread adoption of A/B testing, many assume that a short‑term experiment that shows a positive effect guarantees a sound business decision. In reality, short‑term and long‑term impacts often diverge, and relying solely on short‑term data can lead to erroneous choices.

1. Importance and difficulty of long‑term impact evaluation

Typical A/B experiments run for only 1–2 weeks due to resource constraints and rapid product iteration, yet the effects we truly care about may only become apparent after 1–2 months or longer. Short‑term gains (e.g., more ads increasing immediate revenue) can degrade user experience and reduce long‑term revenue, while new recommendation algorithms may need time to learn and show better results later.

2. Main reasons for short‑term/long‑term inconsistency

User learning effect

Network effect

Delayed experience and data response

Ecosystem changes

These mechanisms are discussed in detail in Chapter 9 of "AB Experiment – Scientific Attribution and Growth Tool".

3. Primary methods for estimating long‑term impact

(1) Long‑cycle experiments

The simplest approach is to extend the experiment duration. The diagram below shows an experiment group (E) and a control group (C) where the difference at time t=1 reflects short‑term impact, and the difference at a sufficiently large time t=T reflects long‑term impact.

This method is costly in traffic and unsuitable for fast‑iteration products, so it is only used for major experiments.

(2) Holdout experiments

After a strategy passes early validation, it is rolled out to the entire user base while keeping a small holdout group (e.g., 5%). The long‑term difference between holdout users and the rest estimates the strategy’s lasting impact.

Many teams stop at these two methods without further exploration.

(3) Post‑analysis and CCD methods

Post‑analysis leverages the insight that if two user groups are identical before and after a period, any difference observed after the experiment period reflects the lingering learning effect of the strategy. The immediate impact is measured at t=1 , while the long‑term learning effect appears at t=T . However, a forgetting effect can cause the groups to converge over time, so measurements must be taken shortly after the strategy is withdrawn.

A refined design creates a new experiment group at each measurement point (e.g., groups E, E1 at T1 , E2 at T2 ) to capture learning effects over time, though it consumes considerable traffic.

(4) Surrogate‑metric prediction method

This approach seeks a universal model that predicts long‑term impact from short‑term experiment results and user behavior, assuming most behavior changes are gradual. The workflow includes:

Define the long‑term target metric Y .

Select surrogate metrics S_i (including Y itself).

Assume the experiment assignment W(i,t) influences current surrogates S(i,t) and target Y(t) , and surrogates influence next‑step surrogates and target, but W(i,t) does not directly affect future steps.

Train separate models for each surrogate (using values up to time E‑1 as features and value at E as label) and for the target (using surrogates at E‑1 as features).

Iteratively predict forward to obtain surrogate values at the desired horizon T , then predict Y(T) . Build models for both experiment and control groups; the difference yields the estimated long‑term lift.

WeChat’s experiment team applied this method to the mini‑program search scenario with promising results (see figures below).

Further details can be found in the referenced papers:

Google research paper

SSRN paper

Feel free to share your data‑science topics in the comments.