How JD’s HotKey Middleware Boosts Backend Performance via Protobuf

Publish

HotKey, based on the 618 stable version 0.2, introduces protobuf serialization and optimizes transmission objects.

The worker’s single‑machine performance increased from a stable 200k QPS (30k peak) to a stable 300k QPS and a peak of 370k QPS, while CPU peak usage dropped by 15%.

The middleware is currently deployed in more than ten core JD departments, serving the JD App backend, middle‑platform, and data middle‑platform.

Architecture

The JD APP backend hot‑data detection framework has undergone multiple high‑pressure stress tests and the 2020 JD 618 promotion. It probes billions of keys daily, accurately captures massive crawler and abusive user traffic, and pushes hot items to all service JVMs within milliseconds, greatly reducing query pressure on the data layer and improving application performance.

Performance metrics focus on two aspects:

Detection performance: an 8‑core worker can handle 160k key detection tasks per second; a 16‑core worker processes over 300k per second stably, reaching 370k in tests.

Push performance: under high‑concurrency writes, the system pushes 100‑120k updates per second stably; in pure‑push mode it can sustain 400‑600k pushes per second, with an 800k‑push peak for a few seconds.

In real scenarios, a 1:1000 ratio (one worker supporting 1000 service instances) can save substantial storage resources, such as reducing the need to expand Redis clusters.

Introduction

The framework detects any sudden hot requests—hot data (e.g., a surge of requests for the same product), hot users (crawlers, bots), hot interfaces (massive requests to the same API)—with millisecond precision, then pushes these hot keys to the JVM memory of all deployed machines. Clients can use the hot keys for local caching, access denial, circuit breaking, or default responses, thereby relieving pressure on backend storage. The hot keys remain consistent across the entire application cluster.

Core function: hot‑data detection and push to all servers.

Applicable scenarios include:

MySQL hot‑data local cache

Redis hot‑data local cache

Blacklist user local cache

Crawler user rate limiting

Interface and user‑level rate limiting

Single‑machine interface/user limiting

Cluster‑wide user limiting

Cluster‑wide interface limiting

UI Effect

Source code & article reference: https://gitee.com/jd-platform-opensource/hotkey