Evolution and Architecture of JD.com Recommendation System

In e‑commerce, recommendation adds value by uncovering users' latent purchase needs, shortening the distance to products, and enhancing the shopping experience.

JD.com’s recommendation journey began in 2012 with rule‑based matching. Early recommendation components were isolated, with no engineering or algorithmic integration. The rise of big data in 2013 forced a redesign, as traditional methods could not keep up with rapid business growth.

With the expansion of mobile internet and multiple screens (JD App, PC site, M‑site, WeChat, QQ, etc.), recommendation types grew from pure product suggestions to activities, categories, coupons, floors, entry images, articles, lists, and curated goods. Strong demand for personalization led to a 2015 system upgrade, and the 2016 “smart marketplace” dramatically boosted GMV, reduced manual costs, and earned a 2016 Group Outstanding Product award.

Recommendation Products

At any point in a user's purchase journey—interest, decision, or order—recommendation products can assist decision‑making.

1. Development Process of Recommendation Products

The evolution moved from simple association to personalized, then to scene‑aware intelligent recommendation, incorporating multiple features, dimensions, real‑time behavior, and contextual information.

Recommendation product development timeline

2. Multi‑Screen, Multi‑Type Product Forms

Recommendation types now cover products, activities, categories, coupons, floors, entry images, articles, lists, and curated goods. Multi‑screen integration collects user behavior via front‑end instrumentation, sends events to a click‑stream system, and processes them in a real‑time computation platform to re‑rank results based on inferred interests.

Recommendation System Architecture

1. Overall Business Architecture

The goal is to precisely model user purchase intent, recommend items they are likely to buy, improve conversion, and increase user stickiness.

Business architecture of the recommendation system

The system provides a unified HTTP recommendation service for all JD terminals.

Model services deliver personalized capabilities through user behavior, user profile, product profile, regional profile, and feature services, simplifying and sharpening personalization.

Machine‑learning stage trains various models, validates them with offline metrics and online A/B tests, and selects the best for conversion improvement.

Data platform collects and processes raw data, which is the foundation for healthy recommendation performance.

2. Personalized Recommendation Architecture

Early versions were simple, with each product as an independent service. The new system is a systematic engineering effort that tightly couples data, architecture, algorithms, and UI/UX. It aims to turn “one‑size‑fits‑all” into “one‑size‑fits‑each”, supporting multiple recommendation types such as products, stores, brands, activities, coupons, and floors.

New personalized recommendation system architecture

Key components include:

Data processing (offline preprocessing, model training, real‑time behavior ingestion, real‑time feature computation).

Recommendation platform (gateway, scheduler, engine) handling request routing, policy‑driven traffic splitting, and debugging.

Personalized foundation services (user/profile, product/profile, behavior, prediction) providing long‑term, short‑term, and real‑time interests.

Feature service platform for declaring, managing, and sharing multi‑dimensional features, supporting rapid A/B experiments.

Personalized techniques (online learning, deep learning) that re‑rank candidates using massive feature sets.

The architecture supports multi‑type recommendations, multi‑screen products, fast A/B iteration, decoupling of storage, engine, and prediction, custom event tracking, and scenario replay.

Data Platform

JD’s massive user base and full‑range catalog generate rich behavior data (browsing, cart, follow, search, purchase, review) and product attributes (brand, category, description, price). This data fuels large‑scale machine learning for precise personalization.

1. Data Collection

User actions on JD App, PC site, WeChat, or QQ trigger click‑stream events, which are streamed in real time and logged for offline processing. Periodic extraction feeds a big‑data platform where data scientists train models that later influence recommendation services, creating a closed data‑collection loop.

2. Offline Computation

Offline jobs (models, features, user/product profiles) run on Hadoop MapReduce and Spark, with results imported via a plug‑in export tool to reduce development and maintenance costs.

Offline computation architecture

3. Online Computation

Online services compute real‑time behavior, profiles, feedback, and interaction features. Data streams from Kafka or JMQ are consumed by Storm or Spark, then stored in Redis and HBase for low‑latency access.

Online computation architecture

Recommendation Engine

The core of personalized recommendation is the recommendation engine, which performs candidate recall, rule filtering, scoring with machine‑learning models, model fusion, and diversification.

Recommendation engine architecture

The engine splits tasks by recall source, balances load, and executes recommenders that handle recall, rule filtering, feature calculation, and ranking.

Recommender architecture

Recall gathers candidates based on user profile, preferences, and region; cold‑start services handle new users. Rule filtering removes illegal or duplicate items. Feature calculation combines real‑time behavior, profiles, knowledge graphs, and feature services to produce vectors. Scoring applies trained models, and the final list is merged, diversified, and may be fine‑tuned for user experience.

Implementation of "You May Like" on JD App

For more JD 618 technical details, see the book "Battle of 618: Exploring JD’s Technology Winning Strategies".

Source: http://www.broadview.com.cn/article/419434

Architecture Digest

ID: ArchDigest

Internet Application Architecture | Architecture Technology | Large‑Scale Websites | Big Data | Machine Learning

More exciting articles, click below to read the original.