How Alibaba Leverages Deep Learning to Revolutionize E‑Commerce Search

Deep Learning in Search Overview

Deep learning, represented by breakthroughs in image, speech, and NLP, has also impacted information retrieval and personalization. Recent works such as Wide&Deep, DSSM, DeepFM, Deep CF, and RNN recommender illustrate the combination of deep and shallow models for recommendation.

Industrial‑grade retrieval or personalization systems require three conditions: strong computational capability, excellent model design, and suitable application scenarios. Senior algorithm expert from Alibaba Search introduces the progress in deep learning systems, algorithms, and search applications.

Four Aspects of Deep Learning in Search

System

Powerful training platforms and online prediction systems are essential. Alibaba unifies offline and online deep learning frameworks and prediction services on TensorFlow, implementing end‑to‑end pipelines for log processing, feature extraction, model training, and deployment, greatly improving iteration efficiency.

Search Applications

Four technical directions—intelligent interaction, semantic search, intelligent matching, and intelligent decision—enable full‑link deep learning upgrades and multi‑scenario, multi‑objective joint optimization.

Performance Optimization

Work on model compression, quantization, low‑rank decomposition, and binary networks lays the foundation for inference performance and hardware‑software co‑optimization.

Ranking Platformization

Unified services for PC, mobile, and shop‑inside search reuse features and models, allowing rapid upgrades across business lines.

Search System Architecture

Key components:

Offline data platform ODPS for log join, feature extraction, and offline model feature generation.

Offline ML platform PAI built on Parameter Server and TensorFlow for parallel training and prediction.

Streaming and online learning platform Porsche for real‑time log parsing, feature join, and online learning.

Online service platform comprising engine, ranking service, and search platform, supporting feature and model reuse across multiple business lines.

Machine Learning Platform and Online Prediction

Search training samples come from user behavior streams. Large models are first pretrained offline, then fine‑tuned online. Both PAI and Porsche use tf‑pai, which optimizes communication, sparse parameter storage, and GPU memory, enabling training on billions of samples and parameters.

Although GPUs are supported, CPU remains dominant in search because features are shallow and low‑dimensional, allowing idle CPUs during traffic lows to be repurposed for training.

Online Real‑Time Prediction (RTP) handles thousands of items per query, using heterogeneous computing, operator fusion, and model sharding; during Double‑11, 550 GPU cards were employed. RTP also provides seamless deployment of offline/online models.

Algorithms

Intelligent Interaction

Search acts as an interactive recommender. Keyword input triggers personalized results; proactive keyword recommendation and dialogue‑style guidance (e.g., “What age is your baby?”) enhance user intent clarification and multi‑turn interaction.

Semantic Search

Bridges the gap between queries and product content via query tagging, rewriting, content understanding, and semantic matching (DSSM, multi‑layer LSTM+attention). Negative samples are generated from knowledge graphs to improve quality.

Intelligent Matching

Includes user perception network (ibrain) with billions of parameters, multi‑modal learning for product features, DeepFM for feature combination, online deep ranking with sample weighting, global ranking using context‑aware click probability, and vector‑based recall engines.

Multi‑Agent Collaborative Learning for Decision Making

Joint learning across heterogeneous scenarios achieves up to 12% improvement in combined metrics during Double‑11, surpassing non‑joint baselines.

Future Plans

General user representation learning beyond query‑centric attention.

Unified query representation for category prediction, rewriting, and recommendation.

Semantic recall and relevance models to curb keyword stuffing.

Search‑chain joint optimization covering query guidance, ranking, and post‑search recommendation.

Cross‑scenario joint optimization for main search and shop‑inside search.

Multi‑objective joint optimization addressing diversity, fairness, and commercialization.

Transforming search into a personal assistant with multi‑turn dialogue and emotional understanding.