Automating Causal Subpopulation Mining: Tencent Music’s Experiment Platform Breaks Down the Process

1. Project Background

In Tencent Music’s product iteration cycle, the experiment platform serves as a standard evaluation tool. After a new feature is launched, a small‑scale vertical random experiment validates its effect, and the data‑science team recovers and analyzes the data to decide whether to promote the feature.

When the overall experiment effect is not significant but certain users show a strong preference, the heterogeneity suggests the need for subpopulation analysis. Identifying and targeting users who respond positively to a strategy requires fine‑grained modeling, which historically involved lengthy manual feature engineering and scheduling.

2. Solution Approach

The platform builds an automated strategy‑positive subpopulation mining tool that reduces the end‑to‑end workflow to a few hours.

Data Preparation

The underlying data consists of three tables:

Assignments – records whether a user belongs to the experiment group and the timestamp.

Metrics – records user behavior metrics during the experiment.

Features – contains static and dynamic user features at the experiment start date.

For each user, the earliest experiment or control group assignment date is extracted as the analysis start point. This date is joined with the Metrics table to aggregate behavior indicators over a post‑intervention period, producing a single record per user. The Features table is then joined to attach the user’s snapshot features on the assignment day. The resulting dataset includes:

X: confounding factors (multi‑dimensional user features at assignment).

t: treatment indicator (1/0).

y: target metric value.

The data is stored as Parquet files in Tencent Cloud COS and converted to Dask CUDF DataFrames for parallel GPU processing.

Conditional Average Treatment Effect (CATE) Estimation

The goal is to predict the expected metric difference between treatment and control for a given user feature vector X. The platform implements several meta‑learner methods:

S‑Learner : treats the treatment indicator t as an additional feature and trains a single model; predictions are made with t=1 and t=0, and the difference yields the uplift.

T‑Learner : trains separate models for treatment and control groups; the difference of the two predictions gives the uplift. This method can suffer when group sizes are imbalanced.

X‑Learner : trains separate models for each group, uses each to predict the opposite group’s outcomes, and combines the two uplift estimates with propensity‑score weighting to mitigate imbalance issues.

Supported base algorithms include XGBoost, LightGBM, and Random Forest, all executed on GPU clusters for speed and generalization.

Representation Learning

Beyond traditional meta‑learners, the platform incorporates representation‑learning approaches such as dual‑head neural networks and DragonNet, which add a propensity‑score head to jointly learn shared representations and improve CATE estimation.

Model Evaluation – AUUC

After training multiple CATE models, the Area Under the Uplift Curve (AUUC) is used to assess performance. The evaluation steps are:

Rank all samples by predicted uplift.

Compute the average metric difference between treatment and control within each percentile bucket.

Plot the uplift curve and calculate its area.

A larger AUUC indicates higher sensitivity to the intervention.

Hyper‑Parameter Tuning – Optuna

The platform leverages Optuna for automated hyper‑parameter search using Bayesian optimization and TPE. The workflow includes defining the model and search space, running multiple training‑validation rounds, evaluating AUUC on the validation set, and iteratively updating the search strategy to obtain the best parameter combination.

3. Result Visualization

Predicted uplift values are combined with user features to build a simplified decision‑tree model that extracts actionable split paths for strategy formulation. This two‑stage modeling (full uplift model followed by an interpretable tree) provides both predictive power and business‑friendly explanations.

Extract the most positive and negative uplift paths.

Summarize split conditions into actionable rules.

Automatically generate a textual report and push it to product managers via enterprise WeChat.

4. Full‑Process Summary

The end‑to‑end pipeline—data preparation, CATE training, evaluation, hyper‑parameter tuning, visualization, and reporting—is fully integrated into the experiment platform. A complete modeling cycle can finish within 30 minutes to an hour, compared with days of manual effort.

Supports multiple causal‑inference models (meta‑learners, representation learning).

Provides built‑in unified dimension tables to avoid manual feature engineering.

Offers a seamless workflow from model training to report generation.

5. Other Causal‑Inference Functions

Double Difference

The platform implements classic difference‑in‑differences to address pre‑experiment imbalances by computing (post‑treatment – pre‑treatment) for both groups and taking the net effect.

Propensity‑Score Matching (PSM)

For non‑randomized experiments, a binary classifier predicts the probability of receiving treatment (propensity score). Users are bucketed by score, and control users are sampled to match the treatment distribution. Matching quality is evaluated using the Standardized Mean Difference (SMD), with SMD < 0.1 indicating acceptable balance.

6. Model Deployment & System Integration

The mined subpopulations are not directly fed back into recommendation models; instead, the uplift insights guide feature engineering and strategy design. Product and operations teams can use the generated rules to target specific user segments in subsequent experiments, creating a closed‑loop optimization workflow.

7. Q&A

Discussion covered model integration with recommendation systems, fine‑grained operational support, and iterative experiment validation using uplift‑identified user groups.