How Propensity Score Matching Unlocks Accurate Causal Effects Without A/B Tests

Introduction

When A/B testing is impossible or insufficient (e.g., due to non‑compliance), correctly estimating causal effects requires removing the influence of confounding variables, and Propensity Score Matching (PSM) is one of the most common methods for doing so.

Why PSM?

Confounders affect both the treatment (intervention) and the outcome, making the observed correlation unreliable for causal inference. By blocking the confounder’s impact, we can obtain a more accurate measurement of the true intervention effect.

Overall Framework

1. Propensity Score Calculation : Build a machine‑learning model to predict each sample’s probability of receiving the treatment (the propensity score). 2. Sample Matching : For each treated sample, find one or more control samples with similar propensity scores (nearest‑neighbor or caliper matching). 3. Effect Estimation : Compare outcomes between matched treated and control groups to estimate the intervention effect.

Key Concepts

Confounders are variables that influence both treatment and outcome (e.g., ability, age, demand). Their presence turns the treatment‑outcome correlation into a spurious relationship.

To eliminate this, we must “block” the confounder so that it becomes independent of the treatment.

When A/B Tests Are Feasible

Randomized experiments (A/B tests) naturally break the link between confounders and treatment, providing the gold standard for causal inference.

When A/B Tests Are Not Feasible

We rely on observational data and apply matching techniques. Matching groups samples with similar confounder profiles, thereby approximating a randomized comparison.

Matching Algorithms

Common algorithms include:

Caliper Nearest‑Neighbor Matching (most used)

Radius Matching

One‑to‑One vs. One‑to‑Many Matching

With‑Replacement vs. Without‑Replacement

Choosing an algorithm involves a trade‑off between bias (systematic error) and variance (estimation variability). More matched control samples reduce variance but may increase bias.

Homogeneity Checks

After matching, verify that the distributions of confounders are similar between treated and control groups using visual plots or statistical tests (t‑test, chi‑square, etc.).

Application Cases

Case 1: Education vs. Income – Higher education correlates with higher income, but ability (a confounder) may drive both.

Case 2: Medication vs. Mortality – Higher mortality among medication users may be due to age, not the drug itself.

Case 3: Event Participation vs. Order Volume – Increased order volume among participants may stem from higher underlying demand.

Driver Activity Evaluation (Real‑World Example)

Scenario: No A/B test for a driver‑incentive campaign during a logistics festival. Steps:

Define treated drivers (participated) and control drivers (did not).

Collect pre‑campaign features (historical orders, GTV, demographics).

Train a model to predict participation probability (propensity score).

Apply caliper nearest‑neighbor matching (1‑to‑1, without replacement).

Check covariate balance between matched groups.

Compare post‑campaign order count and GTV; results show a 31.6% increase in order count and 12.2% increase in GTV for participants.

Summary

This article introduces Propensity Score Matching as a practical causal inference technique for situations where A/B experiments are unavailable or have low coverage. It outlines the full workflow—from propensity score estimation to matching, balance checking, and effect calculation—illustrated with three synthetic examples and a concrete driver‑activity case study.