How Reinforcement Learning Transforms E‑Commerce Search and Recommendation at Scale

1 Search Algorithm Research and Practice

Background Taobao’s search engine must process billions of items within milliseconds for a massive, diverse user base. Traditional Learning‑to‑Rank (LTR) learns from displayed items and cannot guarantee globally optimal rankings, prompting exploration of offline‑online discrepancy solutions, Counterfactual Machine Learning, and online trial‑and‑error methods such as Bandit Learning and Reinforcement Learning.

Earlier experiments applied a Multi‑Armed Bandit (MAB) model to learn ranking policies from user feedback, balancing exploration and exploitation.

Subsequently, a Markov Decision Process (MDP) was introduced to model the search ranking problem, enabling real‑time policy adjustment via deep reinforcement learning. The search engine acts as an agent, the user as the environment, and each ranking decision is a trial‑and‑error step that receives reward from user clicks or purchases.

1.2 Problem Modeling

MDP is defined by the tuple (states, actions, reward function, transition function). The goal is to learn optimal search ranking policies through reinforcement learning.

Four research tasks were pursued:

Tabular RL for price‑tier control (discrete state/action).

Tabular RL for display‑ratio control (discrete state/action).

Value‑function approximation for ranking (continuous state, discrete action).

Policy‑approximation for ranking (continuous state/action).

1.2.1 State Definition

User recent click history is used as state features. In tabular methods states are enumerable discrete variables; in value‑function or policy‑approximation methods states are represented as feature vectors.

1.2.2 Reward Function Definition

The agent receives reward from user actions (clicks, purchases). Reward shaping is later used to enrich the reward signal.

1.3 Algorithm Design

Tabular Method – User price‑tier clicks (0‑7) form the state. An epsilon‑greedy policy selects a price index t, updates the Q‑table based on observed reward, and iterates until convergence.

Q‑learning updates follow the standard formula.

DDPG Method – A linear ranking model is parameterized by a weight vector w for each user state. A deep neural network (Actor) outputs continuous actions (ranking weights). The Actor’s parameters are updated via policy‑gradient using the estimated long‑term reward.

The Actor’s gradient follows the policy‑gradient theorem, and a deep Q‑learning estimator (DQN) is used for value approximation.

1.4 Reward Shaping

Initial reward based only on clicks and purchases showed slow convergence due to limited macro‑signal differences. By incorporating product attributes and prior knowledge (potential function) into the reward, the learning speed improves.

The shaped reward adds a term based on the number of items in the PV and the likelihood of clicks for each item.

1.5 Experimental Results

During the Double‑11 shopping festival, the RL solution was tested on two traffic buckets. The RNEU metric decreased steadily after launch, indicating convergence toward optimal policies, but spiked after midnight due to abrupt user behavior changes.

2 Recommendation Algorithm Research and Practice

2.1 Background

Double‑11 main venue recommendation involves three layers: floors, slots, and material images. The massive scale (millions of images, hundreds of slots) and high QPS (tens of thousands) require careful candidate selection, quota allocation, and model robustness.

2.2 Algorithm Models

2.2.1 GBDT+FTRL Model

GBDT extracts high‑order intermediate features from raw statistics; these leaf‑node features are combined with ID features and fed into a linear FTRL model for CTR prediction.

Key features include user/item ID cross statistics and continuous match‑stage scores.

2.2.2 Wide & Deep Learning Model

Combines a wide linear component (feature crosses) with a deep neural network that embeds categorical features and processes continuous features, producing a unified posterior probability.

2.2.3 Adaptive‑Online‑Learning

Maintains models trained on data from each timestamp, weights them based on evaluation metrics, and dynamically fuses them to adapt to rapid data‑distribution shifts.

2.2.4 Reinforcement Learning for Recommendation

Frames multi‑scene recommendation as a continuous decision problem where the agent selects items (or sets of items) to maximize long‑term cumulative reward. Q‑learning or policy‑gradient methods estimate the expected return Q(s,a) for state‑action pairs.

For single‑item recommendation, the reward is the expected click/purchase likelihood; for multi‑item lists, independence assumptions simplify reward aggregation, though more complex interactions can be modeled.

References

Mnih et al., Playing Atari with Deep Reinforcement Learning, 2013.

Ng et al., Policy invariance under reward transformations, ICML 1999.

Wiewiora, Potential‑based shaping and Q‑value initialization, 2003.