How JD’s API Gateway Handles Tens of Millions of Concurrent Requests

Overview

JD, as an e‑commerce platform, has seen continuous growth in users and business, leading to increasing traffic. To support this, an API gateway was introduced.

The API gateway aims to free both client and server: it standardizes interfaces for clients to reduce integration cost, and shields servers from public‑facing issues so they can focus on business logic, improving development efficiency.

During the recent massive shopping festival, the gateway needed to efficiently handle tens of millions of concurrent requests – the focus of this article.

API Gateway Introduction

The gateway connects clients to backend services and faces high concurrency, security protection, data monitoring, and gray‑release requirements. Core features include high performance, security stability, platform‑wide monitoring, gray‑release capability, and fast integration.

High performance: low latency under high throughput.

Security & stability: authentication, fine‑grained traffic control, real‑time big‑data analysis.

Platform: monitoring, alerting, fault location.

Gray release: gradual rollout by device, user ID, or custom ratio.

Convenient integration: support for HTTP, JSF, mock functions.

Technical Practices

High Concurrency Implementation

The gateway adopts asynchronous processing with NIO multiplexing, turning synchronous handling into asynchronous to maximize request reception.

In synchronous mode, threads are blocked waiting for backend responses, leading to thread‑pool exhaustion and low throughput. Asynchronous processing releases threads immediately, deferring the response until the backend replies, thus allowing continuous request intake.

Security Protection

Fine‑grained traffic control uses a token‑bucket algorithm with queuing or circuit‑breaker strategies, applying per‑endpoint limits based on region, risk level, etc.

Authorization ensures only approved APIs are accessible; unauthorized requests are blocked at the gateway.

Signature authentication uses HMAC‑SHA256 to verify request signatures.

Cross‑origin validation protects web and mini‑program clients from malicious attacks.

Gray Release

For major changes or refactoring, the gateway supports targeted gray testing by device or user ID and percentage‑based rollout, allowing a small portion of traffic to use the new service before full deployment.

Automation Operations

The gateway provides a unified console for independent deployment, rapid scaling, automated configuration, and real‑time monitoring. It enables self‑service API activation, authorization, and dynamic adjustments without service restarts.

Monitoring & Fault Diagnosis

All external requests pass through the gateway, making it an ideal point for collecting metrics such as request volume and latency. The platform offers visual dashboards, alert subscriptions, and real‑time analysis to quickly locate and resolve incidents.

Conclusion

As the sole entry point to backend services, the API gateway is critical for JD’s technology middle platform. It achieves componentization, structured demand, configuration‑driven data, and visualized business, providing high‑performance, secure, and observable access for micro‑service architectures.