JDQ Kafka Bandwidth Control and Rate Limiting Architecture

JDQ, built on the open‑source Apache Kafka message queue, is a high‑throughput, low‑latency, highly reliable real‑time streaming middleware used as JD's unified data bus. The article first examines the shortcomings of Kafka's native throttling mechanisms, especially under massive traffic spikes during major sales events.

2.1 Throttling Overview – Throttling protects brokers from sudden traffic surges that could overload network bandwidth and disk I/O, potentially causing cluster instability or crashes.

2.2 Native Kafka Throttling – Kafka provides two configurable quota parameters (producer_byte_rate, consumer_byte_rate) and three granularity levels (user, client.id, user+client.id). Quotas are stored in Zookeeper under /config paths, and priority is determined by a hierarchy of configuration paths.

/config/users/
/clients/
Highest priority configuration
/config/users/
/clients/
User‑level default for clients
/config/users/
User‑level quota
/config/users/
/clients/
Default user + specific client
/config/users/
/clients/
Default user + default client
/config/users/
Default quota for all users
/config/clients/
Client‑level quota
/config/clients/
Default client quota (lowest priority)

The throttling algorithm calculates a wait time X = (O‑T)/T * W, where O is the observed rate, T the target quota, and W the sampling window (quota.window.num * quota.window.size.seconds).

2.3 Limitations of Kafka Throttling – Issues include broker‑topic level throttling causing rate fluctuations on leader failover, lack of per‑broker throttling and real‑time elasticity, and inability to prioritize traffic based on business importance.

3 JDQ Throttling Core Architecture

3.1 JDQ Throttling Model – Introduces partition‑level throttling so that producer/consumer throughput remains constant even when broker failures occur.

3.2 Multi‑Dimensional Fine‑Grained Throttling – Calculates consumer quota as consumer_byte_rate multiplied by the number of partition leaders for a topic, ensuring total throughput stays at s × m (quota × partition count) regardless of broker failures.

3.3 Single‑Machine and Hierarchical Dynamic Elastic Throttling – Adds a per‑machine bandwidth threshold and importance levels (L0‑L3). When a broker exceeds its bandwidth limit, lower‑priority levels are throttled first, preserving bandwidth for high‑priority services. The system can automatically relax throttling when load drops.

4 Practical Application Results

Experiments validate partition‑level throttling (maintaining 30 MB/s total rate despite a broker failure), hierarchical throttling (gradual reduction of L3, L2, L1 bandwidth while L0 remains unrestricted), and elastic throttling (full throughput restored when bandwidth limits are increased).

5 Future Throttling Optimization Directions

Introduce more granular throttling forms such as QPS‑based and message‑type throttling.

Integrate container‑based elastic scaling to adjust bandwidth limits dynamically.

Develop intelligent throttling planning to automate optimization and reduce manual intervention.

In conclusion, JDQ's enhanced throttling framework addresses Kafka's native limitations by providing multi‑dimensional, dynamic, and priority‑aware rate limiting, improving stability, resource utilization, and operational efficiency for large‑scale real‑time data processing.