Random Bulletin

Jun 17, 2026 · Backend Development

Rebalance Optimization for Million‑QPS Kafka Clusters: From Default to Custom Assignors

At a 5,000‑instance, 20,000‑partition Kafka consumer cluster, the default eager Rebalance caused a 90‑second stop‑the‑world pause, illustrating how a single GC‑induced heartbeat miss can flood brokers with tens of millions of messages; the article dissects eager vs cooperative protocols, built‑in assignor limits, and four custom assignor patterns to scale Rebalance from 100 K to tens of millions of QPS.