How JD Built a Scalable AI-Powered Recommendation Engine for E‑Commerce

In e‑commerce, recommendation creates value by uncovering latent purchase intent, shortening the distance between users and products, and enhancing the shopping experience.

JD's recommendation journey began in 2012 with rule‑based matching, evolved in 2013 with big‑data‑driven designs, and further expanded in 2015‑2016 to support multi‑screen (App, PC, mobile, WeChat) and diverse recommendation types such as activities, coupons, floors, articles, and lists.

1. Recommendation Products

Recommendations assist users at any stage of the purchase decision funnel.

1.1 Development Stages

The product line progressed from simple association to personalized, then to scene‑aware intelligent recommendations, incorporating multi‑feature, multi‑dimensional, real‑time behavior, and contextual signals.

1.2 Multi‑Screen, Multi‑Type Forms

Multiple recommendation types (products, activities, categories, coupons, floors, entry images, articles, lists, good‑goods) are delivered across screens. Front‑end instrumentation captures user actions, which are aggregated by a click‑stream system, processed by a real‑time computation platform, and used to re‑rank results based on inferred interests.

2. Recommendation System Architecture

2.2 Overall Business Architecture

The goal is to model user purchase intent precisely, deliver items users are likely to buy, improve conversion, and increase stickiness.

System Architecture – provides a unified HTTP recommendation service for all JD terminals.

Model Services – shared personalization services (user behavior, user profile, item profile, region profile, feature services) that simplify and improve recommendation accuracy.

Machine Learning – trains various models, validates them with offline metrics and online A/B tests, and selects the best for each scenario.

Data Platform – collects and computes data, the foundation for healthy recommendation performance.

2.3 Personalized Recommendation Architecture

The modern system integrates data, architecture, algorithms, and human‑computer interaction to transform “one size fits all” into “one size per user,” supporting multiple recommendation types and improving conversion, cross‑selling, and user experience.

The architecture separates data processing (offline preprocessing, model training, real‑time behavior ingestion, feature computation) from the recommendation platform (service interaction). Core modules include:

Recommendation Gateway – entry point, request validation, routing, debugging, and response assembly.

Scheduling Engine – policy‑driven traffic distribution, custom event collection, and emergency handling.

Recommendation Engine – implements recall, filtering, feature calculation, ranking, and diversification.

Personalization Base Services – user and item profiles, behavior logs, and prediction services based on machine‑learning models.

Feature Service Platform – declares, manages, and shares multi‑dimensional feature data for rapid experimentation.

3. Data Platform

JD’s massive user base and full‑catalog inventory generate rich behavioral and product attribute data, forming the foundation for large‑scale machine learning and precise personalization.

3.1 Data Collection

User actions on JD’s App, PC site, and WeChat trigger click‑stream events, which are streamed in real time and persisted to logs for offline model training, creating a closed‑loop data pipeline.

3.2 Offline Computing

Offline jobs (models, features, user/item profiles) run on Hadoop MapReduce and Spark; results are exported via a plug‑in tool to a central repository.

3.3 Online Computing

Real‑time pipelines consume Kafka/JMQ streams, process them with Storm or Spark, and store results in Redis and HBase to provide up‑to‑second user behavior, profile, and feature updates.

4. Key Technologies

Key components of the personalized recommendation system include the recommendation engine, feature service platform, and scene‑feature replay.

Recommendation Engine

The engine performs candidate recall, rule filtering, feature computation, scoring, model fusion, and diversification, leveraging machine‑learning models, knowledge graphs, and high‑dimensional features.

Core stages:

Recall – gathers candidates from user profiles, preferences, and region data; cold‑start services handle new users.

Rule Filtering – applies business rules, anti‑fraud checks, and other constraints.

Feature Calculation – combines real‑time behavior, profiles, knowledge graphs, and feature services to build feature vectors.

Ranking – scores candidates with trained models and reorders them according to strategy.

Feature Service Platform

Features (single‑sided like item color, and double‑sided like user‑item interaction) are generated offline or online, managed centrally, and shared across models to reduce development cost and accelerate iteration.

Scene Feature Replay

Online features are captured, sent via HTTP POST to a replay service, stored in a data lake, and later used to retrain models, creating a closed‑loop that aligns offline training with online serving.

5. User Profiling

JD builds comprehensive user profiles covering demographics, shopping habits, and real‑time intent, which feed into personalized floors, flash sales, and other scenario‑specific recommendations.

Conclusion

The JD recommendation system illustrates a full‑stack, AI‑driven approach that evolved from simple association to a multi‑screen, multi‑type, real‑time personalization platform, continuously iterating on algorithms, features, and infrastructure to deliver “one‑person‑one‑face” experiences and drive e‑commerce growth.