How JD.com Scaled Its Order Search with a Real-Time Dual Elasticsearch Cluster

Background

In JD.com’s to‑home order center system, both external merchant order creation and internal upstream/downstream system dependencies generate a huge volume of order queries, leading to a read‑heavy, write‑light workload.

Order data is stored in MySQL, but relying solely on the database for massive queries is impractical, and MySQL lacks support for many complex queries. Therefore, Elasticsearch (ES) is used to handle the primary query load.

ES, a powerful distributed search engine, provides near‑real‑time storage and search. In JD.com’s order system, the ES cluster stores up to 1 billion documents and handles 500 million queries per day.

As JD.com’s business rapidly grew, the ES cluster architecture evolved through several stages, ultimately arriving at a real‑time dual‑cluster backup solution that ensures stable read/write performance.

ES Cluster Evolution

1. Initial Stage

The initial ES deployment was minimal, using default configurations on elastic cloud instances with a chaotic node layout, leading to single‑point failures that were unacceptable for order processing.

2. Cluster Isolation Stage

Mixed‑deployment caused resource contention, affecting order query stability. High‑resource‑consuming nodes were migrated away, and eventually the cluster was moved to dedicated high‑spec physical machines, improving performance.

3. Node Replica Tuning Stage

To maximize resource usage, each ES node was placed on a separate physical machine. To further improve throughput, the replica factor was increased from 1 primary + 1 replica to 1 primary + 2 replicas, and additional machines were added.

The architecture uses a VIP for load balancing. The first layer consists of gateway nodes (ES client nodes) acting as intelligent load balancers, while the second layer contains data nodes that store and process data. Adding an extra replica and expanding machines increased cluster throughput and query performance.

Choosing the appropriate number of shards is a trade‑off: more shards improve single‑ID query throughput but degrade aggregation pagination performance, and vice‑versa. After extensive testing, an optimal shard count was selected.

4. Master‑Slave Cluster Adjustment Stage

To avoid service disruption when a node fails, a standby cluster was introduced. Data is written to both primary and standby clusters (dual‑write). The standby stores recent hot data (≈10% of primary volume) and handles most query traffic, while the primary holds the full dataset for less frequent, full‑order searches.

5. Current Stage: Real‑Time Dual‑Cluster Backup

The primary cluster was upgraded from ES 1.7 to 6.x, requiring index rebuilding. During upgrades, the standby cluster temporarily assumes the primary role to ensure zero downtime. The standby now serves hot data with higher performance, while the primary acts as a cold‑data cluster for full‑order and internal queries.

A one‑click downgrade mechanism allows either cluster to become primary, and the dual‑write strategy adapts accordingly.

ES Order Data Synchronization

Two synchronization approaches were considered:

Listening to MySQL binlog and pushing changes to ES (low coupling but adds a new service and latency).

Directly writing to ES via its API (simpler, low latency, but tightly coupled).

Given the high real‑time requirement, JD.com chose the direct API method. To handle occasional write failures, a compensation mechanism records failed operations in the database; a worker later retries the ES update, ensuring eventual consistency.

Common Pitfalls

1. High‑Real‑Time Queries Use DB

ES refreshes shards every second, providing near‑real‑time visibility. For queries demanding stricter real‑time guarantees, the system falls back to MySQL.

2. Avoid Deep Pagination

Deep pagination (large from values) forces each shard to build large priority queues, consuming CPU and memory. It should be avoided.

3. FieldData vs. Doc Values

Fielddata consumes JVM heap and can cause OOM or latency spikes; switching to doc values (column‑oriented storage on disk) resolves this, and doc values are default from ES 2.x onward.

Conclusion

The rapid iteration of the order center architecture mirrors JD.com’s fast‑growing business. While there is no universally “best” design, the current solution balances throughput, performance, and stability, and future upgrades will continue to push these boundaries.