How JD Built a Scalable AI‑Powered Recommendation System

In e‑commerce, recommendation creates value by uncovering latent purchase intent, shortening the distance between users and products, and enhancing the shopping experience.

Recommendation Products

Recommendation products assist users at any point in the purchase journey, from intent generation to final order.

1. Development Process of Recommendation Products

JD’s recommendation started in 2012 with rule‑based matching. Early products were isolated “tribes” with no engineering or algorithmic overlap. The 2013 big‑data wave and rapid business growth forced a redesign, leading to a new recommendation system.

2. Multi‑Screen, Multi‑Type Product Forms

Recommendation types expanded beyond items to activities, categories, coupons, floors, entrance images, articles, lists, and “good goods.” Multi‑screen integration (App, PC, M‑site, WeChat, QQ, etc.) collects user behavior via front‑end tracking, aggregates it through a click‑stream system, and feeds real‑time preferences into a re‑ranking engine for personalized results.

Recommendation System Architecture

1. Overall Business Architecture

The system aims to model user purchase intent with precise data, recommend items the user is likely to buy, improve conversion, and increase user stickiness.

It provides a unified HTTP recommendation service for all JD terminals.

Model services include user‑behavior, user‑profile, item‑profile, region‑profile, and feature services, simplifying personalized recommendation.

Machine‑learning models are trained offline, validated with A/B tests, and deployed to boost conversion.

The data platform supplies the raw data foundation, crucial for healthy recommendation performance.

2. Personalized Recommendation Architecture

Early versions treated each recommendation product as an independent service. The new system integrates data, architecture, algorithms, and UI to transform “one size fits all” into “individualized for each user,” improving loyalty, decision quality, cross‑selling, and conversion rate. It now supports multiple recommendation types such as items, stores, brands, activities, coupons, and floors.

Data Processing Layer (gray) – offline preprocessing, model training, real‑time behavior ingestion, and feature calculation.

Recommendation Platform (blue) – handles request routing, service interaction, and core modules.

Recommendation Gateway – entry point that validates requests, performs debugging, and assembles responses.

Scheduling Engine – dispatches traffic based on experiment configurations, supports user‑level, random, and parameter‑based splitting, custom tracking, and emergency handling.

Recommendation Engine – implements recall, filtering, feature computation, ranking, and diversification.

Personalized Base Services – user profiles (long‑term, short‑term, real‑time interests), item profiles (product terms, brand, quality, price, gender, age, tags), behavior services (search, click, follow, add‑to‑cart, order), and prediction services trained on historical behavior.

Feature Service Platform – declares, manages, and shares multi‑dimensional features, enabling rapid iteration and A/B testing.

Personalization Technology (yellow) – leverages features and algorithmic models for re‑ranking, using online learning and deep learning for large‑scale feature computation.

The architecture’s advantages include support for multi‑type and multi‑screen recommendations, fast A/B experimentation, decoupled storage and compute, and a platformized feature service.

Data Platform

1. Data Collection

User actions on JD’s platforms (App, PC, WeChat, QQ) trigger click‑stream tracking, which streams data in real time for online consumption and writes logs for offline processing. Logs are periodically extracted to the big‑data center, where machine‑learning teams build models that feed back into the recommendation service, forming a closed‑loop.

2. Offline Computing

Offline jobs run on Hadoop MapReduce and Spark, producing models, features, user and item profiles, and behavior datasets. A plug‑in export tool reduces development and maintenance costs for diverse business scenarios.

3. Online Computing

Online services compute real‑time behavior, profiles, feedback, and interaction features. Data streams from Kafka/JMQ are processed by Storm or Spark, then stored in Redis and HBase for low‑latency access.

Recommendation Engine

The core engine follows a pipeline: recall candidate set, rule filtering, feature calculation, scoring with algorithmic models, model‑fusion ranking, and diversification. It combines machine‑learning models, knowledge graphs, high‑dimensional features, and massive recall to deliver personalized rankings.

Recall Stage – gathers candidates based on user profile, preferences, region, with cold‑start services for new users.

Rule Filtering Stage – applies manual rules, multi‑merchant constraints, coupon rules, etc.

Feature Calculation Stage – merges real‑time behavior, profiles, knowledge graph, and feature services to compute feature vectors for candidates.

Ranking Stage – scores candidates with algorithmic models, reorders based on scores and business policies.

Merge Stage – combines results from multiple recommenders, enforces business rules, and ensures diversity.

Example: JD App’s “Guess You Like” flow – user profile and recent behavior select recall strategies, filter with business rules, extract user/item/cross features, score with models, sort, enrich with reasons, and apply final adjustments for diversity.