Evaluating Long-Term Strategy Effects with A/B Experiments: Causes, Industry Solutions, and Business Cases

The article introduces a research agenda on using A/B experiments to assess the long‑term effectiveness of product strategies, outlining three main parts: the causes of short‑ and long‑term experimental effects, industrial solutions for evaluating long‑term impact, and concrete business case studies.

Reasons for Short‑ and Long‑Term Effects A/B tests are widely used to quantify strategy impact, but limited experiment duration often reveals only short‑term metrics such as immediate DAU spikes, which may be driven by novelty effects. Long‑term outcomes like revenue can lag due to user learning, novelty decay, or seasonal factors. The article categorises these causes into external factors (market equilibrium, seasonality, external events) and internal factors (user learning, novelty decay, primary effects, personalized recommendation biases, sample‑selection bias, and limited observation windows).

Industrial Solutions for Long‑Term Evaluation Seven approaches are described:

User‑learning effect methods – modelling how positive effects amplify over time and using CCD (Cookie‑Cookie‑Day) experiments to isolate long‑term learning.

Personalised recommendation methods – accounting for recommendation system changes that create divergent long‑term and short‑term user experiences.

Short‑term surrogate metric methods – selecting proxy metrics highly correlated with the ultimate “north‑star” metric through correlation analysis and back‑testing.

Surrogate index prediction – regressing multiple short‑term proxies onto long‑term outcomes under unconfoundedness, surrogacy, and comparability assumptions.

Stage‑wise prediction – partitioning time into windows and recursively forecasting future outcomes using past surrogate indices and policy variables.

Observation‑data method – estimating user‑learning effects (novelty and primary effects) via a linear DID‑style model without extra experimental infrastructure.

Population‑bias adjustment – correcting heavy‑user bias in experiments by re‑weighting or jackknife‑style estimators.

Business Case Study The article concludes with a practical scenario where the goal is to maximise matching efficiency (e.g., GMV). When GMV remains unchanged but auxiliary behaviours (clicks, likes, comments) improve, the seven methods are evaluated to determine which best captures the true long‑term impact, highlighting ongoing challenges and the need for further exploration.

In summary, the presented methods provide a toolbox for practitioners to bridge the gap between short‑term experimental observations and long‑term strategic outcomes, while acknowledging each method’s limitations and the importance of context‑specific adaptation.