Unlocking Heterogeneous Treatment Effects: Theory, Methods, and a CATE Tool

Experimental Heterogeneity (HTE)

Experimental heterogeneity, also called Heterogeneous Treatment Effects (HTE), describes the situation where the same treatment yields different effects for different units in an experiment. The Rubin potential‑outcome framework formalises this: for each unit i there exist two potential outcomes Y_i(1) (treated) and Y_i(0) (control). The individual treatment effect (ITE) is τ_i = Y_i(1) - Y_i(0). Averaging over a sub‑population defined by covariates X gives the Conditional Average Treatment Effect (CATE): CATE(x) = E[τ_i | X_i = x].

Interpreting Metric Changes

Before heterogeneity analysis, a metric shift is often interpreted as a uniform effect. After analysing CATE, the effect may diverge across sub‑populations, revealing that a globally negative (or insignificant) result can hide strongly positive effects for specific groups.

Business Significance

Identify how a strategy performs for distinct user segments, guiding iteration and new experiment design.

Discover sub‑populations where a globally ineffective strategy is beneficial, and isolate groups that are harmed.

Model experiment results to enable real‑time targeting of the most responsive audiences.

In a sample of 35 experiments run in June, only about 23 % exhibited detectable heterogeneity from the perspective of the experimental population.

Heterogeneity Analysis Methods Overview

Choosing Splitting Dimensions

The splitting variable X must satisfy the unconfoundedness condition T ⟂ X (treatment assignment is independent of the covariate). When this holds, the conditional expectation of the outcome under treatment equals the unconditional expectation for each value of X, justifying downstream CATE estimation. In practice, use pre‑experiment labels (e.g., the user’s activity level on the day before the experiment starts) to ensure independence.

Common Pitfalls (Bad Cases)

Using post‑experiment activity labels that are affected by the treatment violates T ⟂ X. Fix: use the label from the day before exposure.

Mixing metric‑level dimensions (e.g., product category) with split‑unit dimensions (e.g., user ID). Fix: keep the analysis dimension at the level of the randomisation unit.

When the analysis goal is metric‑level (e.g., SKU conversion), the standard HTE workflow does not apply; instead use metric‑level drill‑down supported by the platform.

Method Selection

Dimension drill‑down : Quick, easy to interpret, but relies on analyst intuition and struggles with interaction effects.

ANOVA / ANCOVA : Provides statistical inference for low‑dimensional interactions; assumes linearity and becomes cumbersome in high dimensions.

Causal Tree : Explores high‑dimensional covariates; may oversimplify complex effects.

Meta‑Learners (S‑Learner, T‑Learner, X‑Learner) : Scalable with modern ML models (e.g., XGBoost); accurate ITE estimates but computationally intensive and lack direct p‑values.

Double Machine Learning (DML) : Unbiased, efficient ITE estimates with confidence intervals; robust to model misspecification but resource‑heavy.

Hybrid ITE + CATE (model + decision‑tree interpreter) : Combines predictive power of ITE models with intuitive tree‑based interpretation; slower than pure tree methods.

CATE Exploration Tool (MVP)

Repository: http://xingyun.jd.com/codingRoot/abtest_ds/CATE_model

The tool automates multidimensional drill‑down, builds a CATE model, and returns the sub‑population with the largest estimated effect.

from CATE_model.utils.workflow import CateWorkFlow
yaml_path = 'config.yaml'  # configure analysis requirements
cate_workflow = CateWorkFlow(yaml_path)
cate_workflow.prepare_analysis()
cate_workflow.execute_cate_auto()
# df_out.styler displays the top CATE sub‑population statistics

Key functionalities:

Automatic retrieval of experiment split‑unit information.

Automatic retrieval of experiment metric data.

Parsing of user‑label tables used for CATE analysis.

Construction of data‑source relationships.

Generation of the sub‑population that maximises CATE for the target metric.

Parameter‑tuning interface for advanced users.

Visualization of model results and feature importance.

Statistical testing and multi‑metric decomposition for identified sub‑populations.

Future Directions

Customisable split‑unit tables.

Integration of custom user‑profile tables with existing profiling pipelines.

Iterative improvement of the CATE model (e.g., hyper‑parameter optimisation).

Template configurations for generic dimensions and specific business scenarios.

Graphical UI to simplify configuration input.

References

Rubin’s causal model and potential‑outcome framework.

ANOVA‑based interaction analysis for CATE.

Meta‑Learner and DML methodologies – see the causalML documentation: https://causalml.readthedocs.io/en/latest/methodology.html

Double Machine Learning and Causal Forest – see Microsoft EconML: https://www.pywhy.org/EconML/spec/overview.html