Data Science Practices in E‑commerce Search: Experimentation, Causal Inference, and Metric Design

Introduction – E‑commerce search generates massive, complex data; it is the core traffic‑distribution and conversion channel, posing many data‑science challenges.

1. Characteristics of the e‑commerce search scenario

Search order attribution is central: the search system links users to merchants across keywords, shop search, coupons, etc., aiming to improve order conversion while balancing relevance and richness. Attribution methods include multi‑touch attribution (first, last, average, Markov, Shapley) and time‑window relevance, acknowledging order‑feedback latency.

2. AB experiment practice

Randomized traffic splitting (Randomization Unit), treatment definition, and metric selection are the three pillars. Units include cookies, device IDs, request IDs; stability issues arise from promotional spikes, requiring AA tests and Multi‑AA tests to detect systematic variance. Sample independence problems (e.g., spill‑over in social apps) are noted.

3. Metric design for experiments

Sensitivity – convergence power, test‑power, required sample size.

Interpretability – ability to decompose and relate to business.

Robustness – resistance to false significance, AA stability.

4. Causal inference foundations

Rubin’s Potential Outcome framework (individual, subgroup, global effects) and Pearl’s Causal Graph are introduced; the latter is costly at billion‑scale click data, used mainly in case studies.

5. Small‑traffic / high‑volatility scenarios

Techniques include propensity‑score matching (PSM) and inverse‑probability‑weighting (IPTW) for sample bias correction, CUPED variance reduction using pre‑treatment covariates, and interleaving (instead of user‑level split) for faster convergence, with caution about position bias.

6. Heterogeneous effect analysis

CATE modeling via Causal Trees to prune insensitive user sub‑groups, and uplift modeling (ITE) with transformed outcomes and tree‑based learners, evaluated by Qini curves.

7. Quasi‑experimental evaluation

When AB is infeasible, methods such as time‑based regression discontinuity (RDDiT) and multi‑difference‑in‑differences (DID with treatment, timing, group) are applied, with emphasis on common‑trend checks and appropriate metric focus (ratio vs. level).

8. Observational metric design and traffic distribution analysis

Search traffic follows a power‑law distribution; metrics include power‑law exponent, top‑80% query coverage, and query entropy. The distribution shows a broken power‑law with saturation at head queries, guiding focus on tail and mid‑tail queries.

Conclusion – From order data construction to AB experiments, causal inference, and metric design, the team leverages advanced data‑science methods to solve real‑world e‑commerce search challenges.