How Reinforcement Learning Transforms E‑Commerce Search and Recommendation

1 Search Algorithm Research and Practice

1.1 Background

Taobao's search engine must handle billions of items with millisecond response times, serving a massive and diverse user base. Traditional Learning‑to‑Rank (LTR) learns from displayed items and cannot exploit unseen samples, prompting exploration of counterfactual learning, multi‑armed bandits, and reinforcement learning.

Previous experiments used a Multi‑Armed Bandit model to learn ranking policies from user feedback, achieving promising results.

1.2 Problem Modeling

We model the search ranking problem as a Markov Decision Process (MDP) <S, A, R, T> where the agent is the ranking system and the environment is the user. States encode recent user clicks, actions correspond to ranking decisions, and rewards are derived from clicks or purchases.

1.3 Algorithm Design

Tabular method : Discretize price buckets (0‑7) as states, use an epsilon‑greedy policy to select a price factor t, and update a Q‑table based on user feedback.

DDPG method : Represent user state as a feature vector, learn a continuous policy (Actor) that outputs ranking weights, and update parameters via policy‑gradient using deep neural networks.

1.4 Reward Shaping

Original reward based only on click and purchase signals proved insufficient for large‑scale Taobao search. We enrich the reward with product attributes using reward‑shaping (potential) functions, improving signal discrimination and accelerating convergence.

1.5 Experimental Results

During the Double‑11 promotion, the RL‑based ranking reduced the RNEU metric over several hours, demonstrating convergence toward an optimal policy, though abrupt user behavior changes at midnight caused temporary performance spikes.

2 Recommendation Algorithm Research and Practice

2.1 Background

The Double‑11 main venue involves a three‑layer recommendation hierarchy (floors, slots, and creative assets). High QPS and diverse presentation formats create challenges for feature learning and bias mitigation.

2.2 Algorithm Models

GBDT+FTRL : Train a Gradient Boosted Decision Tree on raw features to generate high‑order cross features, then feed them into a Follow‑the‑Regularized‑Leader linear model for CTR prediction.

Wide & Deep Learning (WDL) : Combine a wide linear component that learns feature crosses with a deep neural network that embeds categorical features, jointly optimizing the posterior probability of click.

Adaptive Online Learning : Continuously retain models trained on recent logs, weight them by online evaluation metrics, and blend them to adapt to rapid data drift.

Reinforcement Learning for Recommendation : Treat the whole recommendation pipeline as a sequential decision problem, using Q‑learning or policy‑gradient methods to maximize long‑term cumulative reward across multiple scenes.