Optimizing SMS Recall Marketing with Response and Uplift Models: A Ctrip Train Ticket Case Study

Background In the era of diminishing traffic dividends, user retention and recall have become critical for internet companies. Ctrip's train ticket service runs weekly SMS marketing campaigns targeting inactive customers to boost repurchase and user stickiness.

The original rule‑based strategy randomly selected users, resulting in low recall effectiveness and ROI. To address this, the team introduced a two‑stage approach: a Response Model for conversion rate prediction and an Uplift Model for SMS sensitivity estimation.

Problem Definition From a pool of N eligible users, select K users (K < N) for SMS sending under cost constraints to maximize overall conversion rate and ROI.

Solution 1: Response Model A conversion‑rate prediction model (using XGBoost) scores users based on historical SMS logs, selecting the top K users. The experimental design splits users into groups A and B, with A serving as control (random K/2 users) and B as treatment (model‑selected K/2 users). Evaluation metrics include offline AUC and Top‑K recall, and online conversion rate and ROI.

Results showed significant improvements over random selection, but two issues emerged: (a) biased evaluation due to model‑selected users having higher baseline conversion probability, and (b) inability to isolate the incremental effect of SMS because natural conversions were not accounted for.

Solution 2: Uplift Model To capture the true incremental effect of SMS, an uplift model estimates the Individual Treatment Effect (ITE): ITE = P(Y|X=x,T=1) - P(Y|X=x,T=0) . Various uplift learners (Meta‑learner, Tree‑based learner, DNN‑based learner) were considered, with the T‑learner (a type of Meta‑learner) performing best in this scenario.

The uplift approach classifies users into four quadrants: marketing‑sensitive, natural converters, indifferent, and anti‑marketing. The goal is to target the marketing‑sensitive segment to maximize incremental gains.

Experimental Results Two rounds of experiments were conducted. The first validated the Response Model, showing higher conversion for model‑selected users. The second compared the Uplift Model against the Response Model, with offline metrics (Qini Score, AUUC) and online ROI both favoring the uplift approach. Additional offline evaluations on test sets v1 and v2 demonstrated that while models like TARNet, GRF, and PCA+S‑learner performed well, they were sensitive to data distribution shifts; the T‑learner remained stable.

Insights and Future Work The study confirms that uplift modeling is more suitable for intelligent marketing scenarios than traditional response modeling, emphasizing the need for scientifically designed experiments and unbiased sample collection. Future directions include handling multi‑treatment scenarios, improving model robustness, and exploring multi‑objective joint modeling.

References

Künzel et al., Metalearners for estimating heterogeneous treatment effects using machine learning, 2019.

Rzepakowski & Jaroszewicz, Decision trees for uplift modeling with single and multiple treatments, 2012.

Wager & Athey, Estimation and inference of heterogeneous treatment effects using random forests, 2018.

Shalit et al., Estimating individual treatment effect: generalization bounds and algorithms, 2017.

Louizos et al., Causal effect inference with deep latent‑variable models, 2017.

Athey, Tibshirani & Wager, Generalized random forests, 2019.