Huya's Experiment Science Platform: Causal Inference, AB Testing, and Uplift Modeling Practices

Huya, a content‑supply platform, faces typical causal‑inference problems such as estimating new‑user growth from external advertising; the presentation uses this scenario to introduce the platform’s experimental science practices.

The talk explains three levels of causal inference—association, intervention, and counterfactual—and emphasizes the need for scientific measurement and data‑driven decision making.

In advertising‑driven growth, confounding factors (e.g., users who have searched for Huya are more likely to see ads) make simple conversion‑rate calculations biased, and traditional AB tests are limited because external ad exposure cannot be controlled by internal systems.

Huya’s experiment platform has progressed through three stages: establishing an experiment culture, improving experiment efficiency, and expanding the platform’s service boundary to support more complex scenarios.

Metric‑production efficiency is improved by providing reusable metric definitions and a fast production pipeline to reduce the time from experiment need to visible metric.

Experiment verification efficiency focuses on lowering false‑positive rates (type‑I errors) and enhancing statistical power, including better control of multi‑test corrections.

Review efficiency enables users to draw correct conclusions directly from platform results without requiring dedicated data‑engineer support.

Several experiment methods are discussed: multi‑test correction to avoid inflated false positives, the risk of “peeking” at daily significance, and the use of CUPED to increase metric sensitivity.

The Real‑Time API (RTA) is introduced for the “activation” (拉活) scenario, filtering non‑target users (already active or already clicked) to reduce repeat clicks and launch‑rate while keeping core metrics such as ΔDAU stable; adjustments to next‑day retention calculations are described to avoid misleading negative signals.

Service extension is illustrated with uplift modeling for ad‑budget allocation, describing Meta‑Learner (S/T/R/X) and neural‑network approaches, the AUUC evaluation metric, and the need to align AUUC improvements with business‑level KPI changes.

Some business lines (e.g., streamer‑side changes) cannot be tested with AB experiments due to market‑place constraints.

Propensity‑Score Matching (PSM) workflow—score calculation, matching, balance testing, and effect evaluation—is presented, along with challenges when balanced results still yield unstable effects.

A simple attribution method based on weekly activity cycles is shown to separate active‑user influence from baseline patterns, useful for DAU prediction, streamer incentives, and content value assessment.

The summary emphasizes the goal of enabling self‑service experiments, expanding to more complex, high‑value tasks, and continuously enriching the experimental toolbox.

Q&A highlights include distinguishing strategy impact from random variation, sourcing data for uplift models, applying attribution in practice, ensuring fair AA split when monitoring many metrics, leveraging experiment data for targeting and bidding, and key factors that affect ΔDAU conversion rates.