Designing Partitions for Millions of QPS: From Default Settings to Precise Capacity Planning

1. The Promotion Disaster Caused by the Default "3" Partitions

At 1 am during a pre‑sale warm‑up, the order‑event topic suddenly showed 40 k TPS on the producer side while the downstream risk‑control service lag spiked dramatically. The topic had only Partitions: 3 , a default set when the topic was created months earlier for a few thousand TPS. With three partitions, expanding consumer instances to 30 or 300 still leaves only three active consumers, the rest idle waiting for rebalance that never creates a fourth partition.

Attempting to increase partitions online broke the original key‑hashing rule, causing the same user’s events to jump between partitions and destroying strict ordering, which raised the risk of duplicate order detection from a per‑thousand to double‑digit percentages.

Reviewing the Git PR that created the topic revealed a one‑line commit message "Create order-event topic" and an approving reviewer who later left the team.

Key insight: Partition count looks like a simple integer but is actually the deepest well of a message‑queue system; experts calculate it at the well‑head rather than guessing.

The article will explore how partition design evolves from a few thousand TPS to tens of millions QPS, and what engineering lessons each upgrade reveals.

2. Three Partition Approaches Corresponding to Three Scales

Before diving into concrete methods, the article splits "partition design" into three stages, each matching a different traffic magnitude and mindset.

2.1 Stage One – "Default Partitions": Anything That Runs

At a few thousand TPS, the partition count is the most invisible parameter. Creating a topic is a muscle‑memory command: copy‑paste, hit enter, and accept the default of 3 or 6 partitions without debate. The prevailing belief is that partitions are just an integer and the default is sufficient.

This works because each partition handles under 1 k TPS, easily served by a single consumer instance. As long as traffic stays low, nobody questions the setting.

2.2 Stage Two – "Experience Partitions": Match CPU Cores

When traffic reaches 100 k QPS, engineers notice that partition count ties directly to consumer parallelism—Kafka allows only one consumer per partition in a group. A naïve heuristic emerges:

Partitions ≈ Consumer instances ≈ Integer multiple of CPU cores per instance.

This works sometimes but fails at higher scales because it only considers parallelism and treats all other dimensions as constants.

2.3 Stage Three – "Precise Partitions": Capacity‑Planning Model

At million‑plus QPS, partition count becomes an engineering math problem. You must balance throughput, ordering, scalability, fault domain, storage cost, and rebalance overhead. At this stage, partition count is the output of a capacity‑planning model, not a guess.

The three mindset upgrades determine whether a system survives traffic spikes, data‑center failures, or rapid scaling gracefully or collapses instantly.

3. Why the Default Value Is a Trap

Many assume the default is a universally reasonable value. In fact, defaults are designed for the "minimum viable" case, not optimal performance.

Kafka defaults to 1 partition, RocketMQ to 4, Pulsar to 1, and Kinesis requires at least one shard. These defaults only ensure you can send the first message and run a Hello‑World program; they ignore throughput, future scaling, and ordering needs.

3.1 Three Failure Modes of the Default

1. Traffic overwhelms the single partition. A single partition’s write throughput in Kafka ranges from 10 MB/s to 100 MB/s depending on message size, replication factor, and disk type. When traffic exceeds this, producers see buffer buildup, timeouts, and TimeoutException, yet the partition count remains 3.

2. Consumer scaling is ineffective. Adding more consumer instances than partitions creates idle consumers. Expanding to 100 instances still yields only three active consumers because a partition can belong to only one instance at any time.

3. Expanding partitions breaks ordering. Increasing partitions from 3 to 24 changes the hash mapping ( hash(key) % partitionCount), causing keys to jump between partitions and destroying strict ordering.

3.2 The Invisible Cost of the Default

Beyond the explicit failures, the default creates a hidden cost: teams may live with "it works" for months, accumulating billions of messages and dozens of consumer groups before realizing a redesign is needed. The redesign then requires a full‑stack outage, not just a single command.

Default values are cheap to set but expensive to change, especially at tens of millions QPS.

4. Experience Partitions: The CPU‑Core Misconception

When traffic reaches 100 k QPS, a popular rule of thumb spreads:

"Partitions = consumer count = CPU core count."

This sounds scientific but only holds in limited scenarios.

4.1 Three Implicit Assumptions of the Formula

1. Message processing is purely CPU‑bound, with no I/O wait.

2. Processing time variance is negligible, so no hot‑spot skew.

3. Consumer instance count is roughly constant, with no frequent scaling.

In reality, these assumptions rarely hold simultaneously.

4.2 I/O‑Intensive Workloads Make CPU Cores Irrelevant

Consider an order‑processing flow: after receiving an event, the consumer queries Redis for a user profile, calls a risk‑control gRPC service, writes to MySQL, and sends a notification via HTTP. The end‑to‑end latency averages 80 ms, of which only ~3 ms is CPU time; the rest is waiting.

Thus a single CPU core can handle dozens or hundreds of concurrent tasks. Planning partitions solely by core count yields a grossly conservative design with severe throughput shortfall.

4.3 Hotspot Keys Break the Core‑Count Formula

If 70 % of traffic hashes to three partitions (e.g., top merchants), those three consumers become saturated while the remaining 13 are idle. The formula assumes uniform key distribution, which is rarely true for user‑ID, merchant‑ID, or device‑ID keys.

4.4 Elastic Scaling Undermines the Assumption

In containerized environments, consumer instance counts fluctuate daily (e.g., 80 instances by day, 20 at night). If partitions are sized for peak instances, they become under‑utilized at night; if sized for trough, scaling up provides no benefit. The experience formula collapses because a dynamic variable cannot be treated as a constant.

5. Precise Partitions: Treating It as Capacity Planning

At tens of millions QPS, partition design finally evolves from experience to precise calculation. The core belief is:

Partition count is the solution of a system of engineering constraints, not a guessed single value.

5.1 The Five Dimensions of Precise Planning

The article lists five hard constraints that must be satisfied simultaneously:

Increasing partitions improves throughput and parallelism but also raises storage cost, rebalance overhead, and fault‑domain exposure. The optimal solution lies between "just enough" and "over‑engineered".

5.2 Deriving a Lower Bound from Throughput

Assume a peak of 800 k TPS, average message size 1 KB, and a conservative per‑partition write bandwidth of 50 MB/s:

Write volume = 800 k × 1 KB = 800 MB/s.

Minimum partitions = 800 / 50 = 16 . This is only the write‑side lower bound.

On the consumer side, assuming each consumer can handle 5 k TPS and the SLA requires lag ≤ 10 s, the maximum consumer instances is 60. Required parallelism = 800 k / 5 k = 160, but limited to 60 instances, so the partition count must be at least 60. Taking the maximum of the two lower bounds yields ≥ 60 partitions .

5.3 Upper Bound from Ordering Requirements

Too many partitions dilute ordering guarantees. For a strict per‑user order requirement with 80 M daily active users, 60 partitions give ~1.33 M users per partition, while 600 partitions give ~130 k users per partition. More partitions increase metadata size and rebalance time.

Two hidden costs appear:

Metadata bloat: more partitions increase the load on Controllers, ZooKeeper, and Brokers.

Longer consumer rebalance: more partitions mean longer reassignment pauses.

In typical Kafka deployments, total partitions per cluster start degrading beyond 4 000 and become untenable beyond 20 000. Hence a practical per‑topic partition ceiling is 200 ~ 500 .

5.4 The Final Precise Formula

The core formula is:

Final partition count = MAX(throughput lower bound, parallelism lower bound) × 1.5 (redundancy) , capped by the cluster’s metadata tolerance. The 1.5× factor provides a buffer for 6‑12 months of growth, avoiding frequent repartitioning.

6. Ordering: Consistent Hashing Is Not a Silver Bullet

Maintaining strict ordering is the hardest dimension in precise partitioning. Many think adding a key guarantees order, but reality is more complex.

6.1 Two Failure Modes of Simple Key‑Based Partitioning

Problem 1: Expanding partitions breaks the hash mapping. Moving from 16 to 24 partitions changes hash(x) % 16 to hash(x) % 24, causing almost every key to jump to a new partition and breaking order‑dependent logic.

Problem 2: Hot‑key skew. If a merchant ID accounts for 50 % of traffic, its keys concentrate on a few partitions, leading to 15 % overall CPU usage but 90 % CPU on three hotspot partitions.

6.2 Improved Consistent Hashing and Its Limits

Consistent hashing reduces full redistribution to 1/N of keys during expansion, but it does not solve hotspot skew and requires custom producer‑side implementation. In Kafka, you can implement a custom Partitioner interface, but the broker remains unaware of the routing logic; it merely stores messages. Thus consistent hashing is a "soft contract" on the producer side, not a native Kafka capability.

6.3 Hotspot Mitigation: Secondary Sharding + Sticky Routing

Stable high‑QPS systems often combine two techniques:

Secondary sharding: For hot keys, append a random suffix (e.g., shopId + rand(0,10)) to spread them across ten sub‑partitions, then merge on the consumer side. This sacrifices strict order for scalability.

Sticky routing: Maintain a dynamic routing table that sends cold keys through normal hashing and hot keys to dedicated partitions, driven by periodic key‑distribution scans.

Each technique has trade‑offs; reliable ordering requires a holistic engineering system rather than a single algorithm.

7. Scaling Up and Down: Can Partition Count Change?

Assume you have precisely calculated partitions and the system runs for six months, then traffic triples. Expanding partitions is far more complex than it appears.

7.1 Kafka Can Only Expand, Not Shrink

Kafka deliberately disallows partition reduction because shrinking raises semantic issues: what happens to messages in deleted partitions? Offsets? Consumer group migration? No universal solution exists, so Kafka only permits adding partitions.

This means a mis‑chosen partition count can only be corrected by adding more, not by removing.

7.2 Three Hidden Reefs When Expanding

Reef 1: Hash mapping changes. The only mitigation is "pre‑partitioning"—setting the partition count at topic creation to the maximum expected for the next few years.

Reef 2: Rebalance shock. Adding partitions forces all consumers to rebalance, causing pauses from seconds to tens of seconds, which can be a minor incident for latency‑sensitive services like real‑time risk control.

Reef 3: New partitions start with no offset. New partitions begin empty; if consumers start from latest, they miss historical messages; if they start from earliest, they re‑read everything. This creates a trade‑off between data loss and duplicate processing.

7.3 Pulsar’s Bundle Approach

Pulsar treats a topic as a collection of "Bundles" that can split and merge dynamically, bypassing Kafka’s expand‑only limitation. However, this adds architectural complexity and higher operational overhead, making it more suitable for very large‑scale deployments (e.g., 100 k topics) than for small‑to‑mid teams.

8. Fault Domain: Partitions Are More Than Performance Parameters

Beyond performance, partitions define the failure radius.

8.1 Impact of a Single‑Partition Failure

Imagine a 64‑partition topic where one partition’s broker disk fails. The leader switches to a follower, but the partition becomes unavailable for seconds to dozens of seconds. During this window, all messages for that partition fail, error rates spike, and synchronous producers block.

More partitions reduce the impact area of a single‑partition failure; fewer partitions amplify it.

8.2 Deriving a Minimum Partition Count from Fault‑Domain Considerations

Rule of thumb: a single partition should carry no more than 5 % of total traffic. For 100 k TPS, the minimum partition count is 20. This ensures that even if a partition fails, the overall error rate stays below 5 % and SLA remains intact.

8.3 Cross‑Datacenter Layout

In multi‑active deployments across three data centers, partitions must be evenly divisible by the number of zones to enable uniform distribution. This constraint often forces partition counts to odd numbers like 60, 90, or 120, which signal that multiple engineering constraints have been considered.

9. Storage and Metadata: The Overlooked Upper Limit

9.1 Per‑Broker Partition Upper Bound

Each Kafka partition creates a directory with files such as .log, .index, .timeindex, and leader-epoch-checkpoint. Brokers must open all these files, consuming file handles and page cache per partition.

Empirically, a broker starts degrading around 2 000 partitions, showing longer startup times, amplified rebalance oscillations, and soaring ProduceRequest tail latency. Beyond ~4 000 partitions, stability collapses.

9.2 Cluster‑Wide Partition Upper Bound

The total partition count stresses the Controller. In ZooKeeper mode, performance bottlenecks appear beyond 100 k partitions; in KRaft mode the limit rises to the millions, but storage and startup costs increase proportionally.

9.3 Topic Explosion and Partition Budget

At tens of millions QPS, the number of topics can explode from dozens to thousands. If each topic receives 100 partitions, the cluster could face 300 k total partitions—far beyond practical limits.

Therefore, allocate partitions per topic based on scale: small topics get ~8 partitions, large ones up to 256, avoiding a one‑size‑fits‑all approach.

10. From "Default" to "Precise" – Implementation Path

The principles above can be condensed into a two‑step workflow: first confirm the traffic scale (avoid over‑design for low traffic), then, for sufficient scale, solve a multi‑dimensional equation.

10.1 Three Engineering Recommendations

Standardized topic‑creation template: Require fields like estimated TPS, message size, ordering requirement, and consumer instance count; auto‑compute recommended partitions.

Topic‑level capacity water‑mark monitoring: Alert when per‑partition throughput approaches limits, consumer lag exceeds thresholds, or total cluster partitions cross warning lines.

Annual partition audit: Review high‑traffic topics yearly, compare actual traffic and consumer counts against original calculations, and decide on expansion or sharding.

10.2 A Repeated Core Conclusion

Partition count is not a parameter; it is the result of capacity planning. Defaults get you started, precise calculations keep you stable. Below a million QPS this may seem excessive, but at tens of millions it is hard‑earned experience.

11. Final Thoughts: From a Single Value to System Engineering

Returning to the 1 am incident: if the topic had been precisely sized at 60 partitions, if a water‑mark alarm had triggered at 200 k TPS, and if a quarterly topic review existed, the outage likely would not have occurred.

Partition design encapsulates the evolution of engineering thinking from "just run" to "experience‑based" to "capacity‑planned" to "system‑engineered". Each upgrade adds a new dimension rather than discarding the previous one.

When asked, "How many partitions should this topic have?" answer that it is not a fill‑in‑the‑blank question but a system of equations requiring simultaneous consideration of throughput, ordering, scaling, fault domain, and storage cost.