Experiment Design in Two‑Sided Markets – Key Insights from Tencent Advertising Live Session

To broaden participants' technical perspective, Tencent Advertising organized a “Tech‑High” live series, and on July 8, Zhu Zhihua, head of the measurement analysis group, presented “Experiment Design in Two‑Sided Markets” and answered audience questions.

Q: How to select control variables for CUPED variance reduction? A: Control variables should be pre‑treatment variables correlated with the metric; stronger correlation yields better variance reduction, but unrelated variables are harmless. Suitable pre‑treatment dimensions include product_type, site_set, optimization_goal, industry_id, product_id, is_ocpa, and is_two_stage.

Q: How to compute the expectation of a control variable? A: Use the day's overall platform data to calculate the average distribution of the variable, which serves as its expectation.

Q: After applying Bayesian smoothing, which weighting method (MH, click‑based, or spend‑based) should be used for bias calculation? A: The choice of weighting method is independent of Bayesian smoothing; they can be applied separately.

Q: Please explain Bayesian smoothing and how to determine the alpha and beta parameters. A: Bayesian smoothing sets an empirical prior for CVR and updates it with current clicks and conversions. Alpha and beta are the two parameters of the Beta distribution, derived from historical data to reflect the prior belief.

Q: What is the rationale behind the bias metric under probability, and why is it defined in this form? A: Since only predicted (p) and observed (o) values are observable, the metric f(p, o) is designed so that its expectation E[f(p, o)] reflects model bias. The metric follows three criteria: (1) lower observed bias implies lower model bias, (2) E[f(p, o)] depends only on model bias e/p − 1, and (3) E[f(p, o)] is minimal when model bias is zero. Mathematical derivation under these criteria yields the final form.