How JD.com Built a Multi‑Screen Personalized Recommendation Engine

With rapid business growth and the rise of mobile internet, JD.com integrated multiple screens (App, PC, M‑site, WeChat, QQ, etc.) and expanded recommendation types beyond products to include activities, categories, coupons, floors, entry images, articles, lists, and quality goods. Leveraging big data and personalized algorithms, the team upgraded the recommendation system at the end of 2015, achieving notable GMV gains, reduced labor costs, and improved traffic efficiency during the 2016 618 event, earning the 2016 Group Excellent Product award.

1 Recommendation Products

Throughout the purchase journey—from intent to decision to order—recommendations at any touchpoint help users make decisions.

1. Development of Recommendation Products

The evolution progressed from simple association to personalized, then to scene‑aware intelligent recommendations, moving from related/similar items to multi‑feature, multi‑dimensional, real‑time behavior, and contextual recommendations.

2. Multi‑Screen, Multi‑Type Product Forms

Recommendation types now cover products, activities, categories, coupons, floors, entry images, articles, lists, and quality goods. In the mobile‑internet era, integrating user data across screens enables more precise personalization. Front‑end instrumentation captures user actions, which are collected by a click‑stream system, processed in real‑time to infer interests, and then re‑ranked for personalized results. JD.com’s multi‑screen terminals are shown below.

2 Recommendation System Architecture

1. Overall Business Architecture

The system aims to model user purchase intent with comprehensive data, recommend items the user is likely to buy, enhance experience, increase conversion, and strengthen user loyalty.

It provides a unified HTTP recommendation service for all JD terminals.

Model services include user behavior, user profile, product profile, regional profile, and feature services, simplifying and improving personalization.

Machine‑learning models are trained offline, evaluated with A/B tests, and deployed to boost conversion.

Data platforms collect and compute the foundational data essential for healthy recommendation performance.

2. Personalized Recommendation Architecture

Early versions had isolated recommendation services. The new system integrates data, architecture, algorithms, and UI. It transforms “one size fits all” into “individualized for each user,” enhancing loyalty, decision quality, cross‑selling, and conversion rates. It supports multiple recommendation types such as products, stores, brands, activities, coupons, and floors.

Different colors in the architecture diagram represent various processing scenarios:

Data processing (gray) includes offline preprocessing, model training, real‑time behavior ingestion, and feature computation.

Recommendation platform (blue) shows interactions among service modules during request handling.

Recommendation gateway validates requests, distributes them, provides debugging, and assembles responses.

Scheduling engine routes traffic based on experiment configurations, supports user‑based, random, or parameter‑based splitting, custom event collection, and emergency handling.

Recommendation engine performs recall, filtering, feature calculation, ranking, and diversification.

Personalized base services include user profiles (long‑term, short‑term, real‑time interests), product profiles (keywords, quality scores, price tiers, tags), user behavior (search, click, follow, add‑to‑cart, order), and prediction services that adjust recall weights using trained models.

Feature service platform manages feature declaration, sharing, and A/B testing to accelerate personalization iteration.

Personalization technology (yellow) leverages features and algorithmic models for re‑ranking, employing online learning and deep learning for large‑scale feature computation.

The system’s advantages include multi‑type and multi‑screen support, rapid A/B experimentation, decoupled architecture and algorithms, scalable storage and compute, and customizable event tracking.

3 Data Platform

JD’s massive user base and product catalog generate extensive behavior data (browsing, cart, follow, search, purchase, review) and item attributes (brand, category, description, price), forming the foundation for large‑scale machine learning and precise personalization.

1. Data Collection

User actions on JD platforms trigger click‑stream events, which are sent for real‑time processing and logged for offline modeling. Logs are periodically extracted to the big‑data center, where algorithms are trained and applied back to recommendation services, creating a closed data loop.

2. Offline Computation

Offline tasks include model training, feature generation, user and product profiling, and behavior aggregation, primarily running on Hadoop MapReduce with some Spark jobs. Results are exported to storage via a plug‑in tool to reduce development and maintenance costs.

3. Online Computation

Online processing handles real‑time user behavior, profiles, feedback, and feature calculation. It consumes Kafka or JMQ streams via Storm or Spark, storing results in Redis and HBase for immediate use in recommendations.

4 Recommendation Engine

The core of the personalized system includes candidate recall, rule filtering, model scoring, fusion ranking, and result diversification, leveraging machine‑learning models, knowledge graphs, and high‑dimensional features to deliver tailored shopping experiences.

The engine orchestrates task distribution, executes recommenders, and merges results, ensuring load balancing and adherence to business rules.

Recall gathers candidates based on user profiles, preferences, and region; cold‑start services handle new users.

Rule filtering removes items violating business constraints.

Feature calculation combines real‑time behavior, profiles, knowledge graphs, and feature services to produce feature vectors for candidates.

Ranking scores candidates using models and re‑orders them according to strategy.

Merging consolidates results from multiple recommenders, applying diversity rules.

For example, JD.com’s “You May Like” on the App homepage follows this pipeline: select recall methods based on user profile and recent actions, filter by business rules, extract user, item, and cross features, score with models, sort, enrich with recommendation reasons, and perform final adjustments for diversity.

For more technical details on JD.com’s 618 success, see the book “Battle of 618: Unveiling JD.com’s Winning Technology”.