Evaluating Long-Term Effects of Strategies with A/B Experiments: Methods and Case Studies

The article introduces the problem of A/B experiments only detecting short‑term effects due to limited experiment duration, using UI design and revenue examples to illustrate how short‑term gains may not persist.

It explains two broad categories of causes: external factors such as market equilibrium, seasonality, and sudden events; and internal factors like user learning effects, novelty decay, primary effects, and personalization bias, which can lead to mis‑estimated long‑term outcomes.

Seven practical solutions from industry are then described:

User Learning Effect Method: Quantifies how positive effects amplify over time while negative effects fade, exemplified by Google’s CCD (Cookie‑Cookie‑Day) experiment that isolates long‑term learning from short‑term spikes.

Personalized Recommendation Method: Accounts for changes in recommendation systems that cause divergent experiences between long‑term and short‑term groups, using causal graphs to separate strategy, system state, and user preference influences.

Short‑Term Proxy Metric Method: Selects short‑term surrogate metrics highly correlated with the ultimate “north‑star” metric, following a three‑step process of candidate selection, correlation analysis, and back‑testing.

Surrogate Index Prediction Method: Regresses multiple short‑term proxies against the long‑term target, assuming unconfoundedness, surrogacy, and comparability to ensure valid predictions.

Staged Prediction Method: Divides the timeline into windows, recursively predicting future outcomes from past proxies, strategy, and user covariates under a shared‑distribution assumption.

Observation‑Data Method: Models user learning as a linear combination of fixed strategy impact and learning effect, using difference‑in‑differences to obtain unbiased estimates.

Heavy‑User Bias Adjustment Method: Corrects for over‑representation of frequent users in experiments by applying jackknife‑style estimators or re‑weighting sub‑populations.

A concrete business case is presented where matching efficiency is measured by GMV and auxiliary user actions; the seven methods are evaluated, highlighting their respective limitations and the ongoing search for optimal long‑term evaluation solutions.

The article concludes by encouraging practitioners to choose or combine suitable methods based on their specific scenarios.