Uplift Modeling: Quantifying Heterogeneous Treatment Effects at Kuaishou

Introduction – Uplift modeling aims to quantify the differential impact of a treatment across user groups, enabling targeted strategies. The talk outlines Kuaishou's work on uplift model development, application, iteration, and the challenges encountered.

The basic idea is to estimate each individual's response to an experiment rather than the average treatment effect (ATE). For example, a discount coupon may affect female users positively while having little effect on males, making the overall ATE insignificant despite strong subgroup effects.

Traditional segmentation struggles with high‑dimensional heterogeneity, so uplift modeling focuses on estimating heterogeneous treatment effects (HTE) instead of ATE.

Methodology Overview

Introduction

Challenges

Applications

Summary

Estimating HTE – Linear models can be extended by adding interaction terms between treatment and features, turning a constant effect into a function of covariates (CATE). This approach is easy to understand and allows statistical testing via the delta method.

Challenges

1. Continuous treatment variables : In practice, treatment intensity (e.g., number of push notifications) varies, requiring models that capture marginal effects that change with dosage.

2. Statistical inference for nonlinear models : Traditional delta‑method variance estimates break down for complex machine‑learning models, necessitating alternative significance testing methods.

Solutions at Kuaishou

To address continuous treatments, a polynomial or other nonlinear function can be fitted so that the conditional average treatment effect (CATE) varies with treatment level, reflecting diminishing returns.

For nonlinear models, Kuaishou built a dual‑neural‑network architecture (based on Farrell 2020) that constructs interaction features via a first network, then combines them with treatment‑specific functions in a second network to output HTE.

Model evaluation uses three metrics:

BLP (Best Linear Predictor) – measures how much of the estimated HTE is signal versus noise.

GATES (Grouped Average Treatment Effects) – tests significance of HTE across impact‑based groups.

CLAN (Characteristics of Least/Most Impacted Units) – assesses feature importance for HTE.

Applications

1. Fan growth and content production : The model predicts a marginal‑benefit curve for each creator, indicating when additional followers no longer boost production, helping allocate resources and budget.

2. Push notifications : By estimating HTE for different numbers of pushes per device, the model identifies users who still benefit from additional pushes versus those who have reached saturation.

Summary

The dual‑neural‑network + HTE framework helps Kuaishou:

Plan resources by estimating revenue ceilings.

Adjust pricing strategies based on marginal returns.

Perform post‑analysis to ensure model robustness, compare across models, and verify significant features.

Q&A Highlights

Q1: The analysis uses offline data, not A/B test results. Q2: Meta‑learners are fast for online use but less flexible for continuous treatments; Kuaishou designed a method to bridge discrete and continuous scenarios. Q3: Model accuracy is evaluated with train‑test splits and RMSE. Q4: For confounder selection, include as many covariates as possible and check residual correlations in the double‑machine‑learning framework.

Overall, the dual‑neural‑network approach enables precise uplift estimation for heterogeneous user groups, supporting better decision‑making in resource allocation and user engagement strategies.