How Reinforcement Learning Powers Interactive Search in E‑Commerce

In the era where time equals money, reducing search time cost and quickly finding target products is crucial. Modern e‑commerce platforms use intelligent recommendation, and reinforcement learning (RL) can help users and platforms interact to locate desired items efficiently.

Problem Definition & Related Work

Problem Definition

We aim to maximize user‑system interactions, which increase page views (PV), user dwell time, and ad revenue. In intelligent voice interaction, Microsoft uses the CPS metric (conversation turns) with values around 23 for XiaoIce and below 3 for Siri/Google Now. Our goal is to optimize CPS.

The task can also be viewed as a task‑oriented dialog system where users complete shopping with minimal interaction, but we focus on increasing interaction.

Significance of RL in Interactive Search

RL excels at modeling sequential decision problems with delayed rewards, such as sacrificing immediate gain for long‑term benefit. Interactive search is a sequential decision process where the system may forgo the highest immediate click‑through rate (CTR) to present items that increase overall user satisfaction and interaction rounds.

Modeling Interactive Search with RL

We model the scenario as an RL problem where the agent is the recommendation service and the environment includes the user and platform factors. The user submits a query (e.g., "phone"), the agent selects an attribute (e.g., "brand"), and the page displays possible values (e.g., "Huawei", "Xiaomi"). The user may select, deselect, or page‑turn, leading to the next state. This loop forms an episode.

State design considers user demographics, user and agent histories, query embeddings, and static tag scores. Actions correspond to category attributes (brand, material, etc.). The reward is +1 if the user does not exit, otherwise 0, encouraging longer interaction sequences.

System

We implement the RL solution using PAI TensorFlow and the Ali AI Agent (A3gent) DQN component. The neural network accepts multi‑modal inputs (sparse/dense, fixed/variable length, various data types) and shares parameters between embedding and output layers for attribute actions.

Training

We pre‑train on historical data to obtain a reasonable initial policy. Offline DRL improves query‑level CTR by 1.9% over a random baseline, while a statistical ensemble yields 6.8% gain. However, the reward design does not significantly boost CTR, though average interaction rounds increase by 0.16 per session.

Real‑time training is approximated by hourly offline parsing of PV logs to generate episodes, due to limited data volume.

Online Deployment

The DII platform (online algorithm service) integrates TensorFlow model inference. Model updates occur when a new index is detected, taking about half an hour for a 1.4 GB model. This latency is acceptable given the hourly data accumulation in the replay buffer.

Results

Evaluation Methods

Simulator: generate a virtual environment from real data and evaluate average reward.

Human testing: limited manual interaction to collect average reward.

Online testing: deploy in production and monitor reward over time.

Online Effect

After training, CPS improves noticeably compared to the ensemble version. Average user interaction rounds increase by 1.5 (over 30%). Tag CTR shows little change because the reward encourages interaction rather than immediate clicks.

Conclusion & Outlook

RL‑based task‑oriented dialogue systems have succeeded in customer service and medical diagnosis. In e‑commerce, the massive action space (up to millions of dimensions) poses a significant challenge. Future work includes hierarchical RL, better information sharing across categories, efficient online exploration, and stable experience replay for rapid learning.

References

End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning

A network-based end-to-end trainable task-oriented dialogue system

Learning end-to-end goal-oriented dialog

A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue.

An end-to-end trainable neural network model with belief tracking for task-oriented dialog.

End-to-end task-completion neural dialogue systems

Iterative policy learning in end-to-end trainable task-oriented neural dialog models

Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning

Towards end-to-end reinforcement learning of dialogue agents for information access