How to Evaluate Policies Without Experiments: Synthetic Control, Matching, and Causal Impact Explained

This article is the sixth chapter of the "Trusted Experiment Whitepaper" series, focusing on observational research methods such as Synthetic Control Method, Matching Methods, and Causal Impact, and their application in Meituan's delivery business where controlled experiments are infeasible.

Chapter Contents

6.1 Synthetic Control Method

6.2 Matching Methods

6.3 Causal Impact

6.4 Outlook and Extensions

6.1 Synthetic Control Method

6.1.1 Overview

When a policy (e.g., Beijing's 2024 green packaging requirement) affects only a specific region, a synthetic control group is constructed by linearly weighting similar untreated regions to estimate the counterfactual outcome.

6.1.2 Principle

The method learns weights from pre‑intervention panel data to create a weighted “synthetic” control that mimics the treated unit. The average treatment effect on the treated (ATT) is estimated by comparing the post‑intervention outcomes of the treated unit and its synthetic counterpart.

Key assumptions include high‑quality multi‑period panel data, sufficient untreated units, and similarity between treated and synthetic control before intervention.

6.1.3 Case Study

Meituan evaluated a new city‑wide operation strategy by constructing a synthetic control from other cities that had not yet adopted the strategy, allowing impact assessment without a randomized trial.

6.2 Matching Methods

6.2.1 Overview

Matching balances covariate distributions between treated and control groups to reduce selection bias, especially when random experiments are prohibited by law or cost.

6.2.2 Principle

Matching relies on the Conditional Independence Assumption (treatment assignment independent of potential outcomes given covariates) and the Overlap Assumption (both treated and control units have positive probability of receiving each treatment). Common distance metrics include propensity‑score distance, Mahalanobis distance, and exact matching.

Propensity scores are estimated via logistic or probit regression, then nearest‑neighbor or caliper matching is performed. Balance is evaluated using standardized mean differences (SMD) and distribution plots.

6.2.3 Practical Case

Meituan's "full‑city gray" strategy was evaluated by matching users who purchased a coupon package with similar non‑purchasers, using propensity‑score matching to estimate the incremental order volume.

6.3 Causal Impact

6.3.1 Overview

Causal Impact uses Bayesian Structural Time Series (BSTS) to construct a virtual control series, enabling impact assessment when randomized experiments are impossible.

6.3.2 Principle

The BSTS model combines a design matrix, latent state vector, and observation/transition noise. Bayesian inference (often via MCMC) yields posterior distributions of counterfactual predictions, from which pointwise and cumulative causal effects are derived.

6.3.3 Practical Case

For a city‑wide integrated marketing campaign, Causal Impact was applied by selecting untreated cities as donors, incorporating weather and other exogenous variables, and estimating the campaign’s lift on order volume.

6.4 Outlook and Extensions

Beyond the three core methods, the article mentions inverse‑probability weighting, doubly robust estimation, instrumental variables, and double machine learning as advanced tools for observational causal analysis.

References

Abadie & Gardeazabal (2003)

Rubin (1983)

Stuart (2010)

Ho et al. (2007)

Brodersen et al. (2015) – Causal Impact

etc.