JDQ Kafka Bandwidth Throttling Architecture and Optimization

JDQ, built on the open‑source Apache Kafka message queue, serves as JD's unified real‑time data bus with high throughput, low latency, and high reliability. The article examines the shortcomings of Kafka's native throttling in handling massive data flows during peak events such as 618 and 11.11.

Background : As digital transformation drives massive data generation, enterprises need efficient bandwidth control to maintain service stability without adding excessive resources.

2.1 Throttling Overview : Throttling protects servers from traffic surges that could overload broker bandwidth and disk I/O, potentially causing cluster instability.

2.2 Native Kafka Throttling Mechanism : Kafka provides two configurable quota parameters— producer_byte_rate and consumer_byte_rate —and three granularity levels (user, client.id, user+client.id). Quota configurations are stored in ZooKeeper under /config paths, with hierarchical precedence.

Quota precedence (high to low):

/config/users/<user>/clients/<client-id>     Specific user+client-id configuration (highest priority)
/config/users/<user>/clients/<default>       User configuration with client defaults
/config/users/<user>                         Individual user quota
/config/users/<default>/clients/<client-id>  Default user quota with specific client-id
/config/users/<default>/clients/<default>    Default user quota with default client-id
/config/users/<default>                      Default quota for all users
/config/clients/<client-id>                  Specific client-id quota
/config/clients/<default>                    Default quota for all clients (lowest priority)

The throttling algorithm computes a wait time X = (O‑T)/T * W, where O is the observed rate, T the target quota, and W the sampling window.

2.4 Limitations of Native Kafka Throttling : (1) Broker‑topic throttling is vulnerable to leader failover, causing throughput drops; (2) Lack of per‑broker throttling and real‑time elasticity; (3) No support for business priority differentiation.

3 JDQ Throttling Core Architecture

3.1 JDQ Throttling Model : Introduces partition‑level throttling, ensuring total producer/consumer rates remain constant regardless of broker failures.

3.2 Multi‑Dimensional Fine‑Grained Granularity : Adds partition‑level limits and calculates consumer quotas as consumer_byte_rate * number_of_leader_partitions , preserving aggregate throughput even when leaders shift.

3.3 Single‑Machine and Hierarchical Dynamic Throttling : Implements bandwidth thresholds per broker and priority levels (L0‑L3). When a broker exceeds its threshold, lower‑priority traffic is throttled first, allowing high‑priority services uninterrupted bandwidth.

4 Practical Application Effects : Experiments demonstrate that partition‑level throttling maintains a total consumption rate of 30 MB/s despite a broker failure, and hierarchical throttling dynamically allocates bandwidth according to business importance during traffic spikes.

5 Future Throttling Optimization Directions : Plans include multi‑form, multi‑granularity throttling (e.g., QPS‑based), container‑native elastic scaling, and intelligent automated throttling strategies.

Conclusion : JDQ’s enhanced throttling architecture addresses the scalability, elasticity, and priority challenges of native Kafka, providing a robust solution for real‑time data processing in the big‑data era.