From Random to Intelligent: Partition Selection Strategies for 10 Million QPS Systems

Partition Is Not a Simple Bucket

In the era of single‑node queues a topic is a single linear stream, but distributed systems split a topic into parallel sub‑queues (partitions) so producers and consumers become a bundle of pipes. Partition selection therefore influences throughput, ordering, balance, and hotspot handling.

The diagram hides four relationships: how producers distribute, how partitions carry data, how consumers subscribe, and how order is guaranteed. Partition selection is the rule that ties these four strings together.

Partition selection is not just a delivery endpoint; it is the rule that allocates parallelism and ordering semantics.

Understanding this explains why a trivial strategy works at small scale but fails dramatically at large scale.

Real Pressure at 10 Million QPS

Moving from 100 k to 1 M QPS mainly stresses raw throughput; moving from 1 M to 10 M QPS stresses stability under extreme conditions. Four concrete problems appear:

Hotspot concentration : a few high‑frequency keys (e.g., top streamers, hot products) may hash to the same partition, saturating its disk and network.

Batch efficiency loss : random placement scatters a producer’s batch across many partitions, leaving each buffer with only a few messages; the resulting “tiny‑packet ocean” inflates request count and reduces compression.

Consumer skew : uneven data volumes cause a few slow consumers to hold back the whole consumer group.

Fault amplification : if a broker’s leader becomes temporarily unavailable, random routing continues to send traffic there, magnifying failures.

First Generation Strategy: Naïve Random and Round‑Robin

Early Kafka (0.8‑2.4) used a default that chose a random partition when the key was null and a round‑robin counter for each producer instance otherwise. Random selection picks a partition via a random number; round‑robin increments a counter and cycles through partitions. At low scale, both appear balanced.

The strategy breaks in two scenarios:

Batch sending gets scattered : a producer generating 10 k messages per second targeting 64 partitions would place on average only 150 messages per partition per second. With default linger.ms and batch.size, each flush sends a tiny batch, inflating RPC count and reducing compression.

Failure‑induced indiscriminate hits : if a partition’s leader briefly disappears, random routing still sends a fraction of traffic there, causing a cascade of retries and amplified load on the unhealthy partition.

Random and round‑robin assume equal partitions, moderate partition count, and uniform messages—assumptions that collapse at 10 M QPS.

Second Generation Strategy: Key Hash for Order

To guarantee that messages for the same business object (order ID, user ID, device ID) are consumed in order, producers hash the key and mod by partition count. This ensures per‑key ordering but shifts the balancing problem to key distribution.

If keys are uniformly random (UUID, Snowflake ID) the hash behaves like random. In most real workloads, long‑tail distributions cause a few hot keys to dominate traffic. The article cites a real e‑commerce promotion where the top 100 users generated 30 % of order events, overwhelming a few partitions.

Another hidden pitfall is partition‑count change : increasing the number of partitions changes every key’s hash result, breaking the ordering guarantee and forcing either temporary disorder or complex dual‑write migration. This is why the Kafka community discourages arbitrary online partition expansion.

Key hash solves ordering but introduces hotspot amplification, immutable partition count, and inability to adapt to real‑time load.

Third Generation Strategy: Sticky Partitioning

Kafka 2.4 introduced sticky partitioning as the default for messages without a key. Instead of picking a new partition for each record, the producer sticks to a single partition for the whole batch; after the batch is flushed, a new partition is chosen.

This change turns the probabilistic “pick‑per‑record” into a deterministic “pack‑per‑batch”, allowing larger batches, higher compression, and fewer network requests, which dramatically improves throughput.

Sticky partitioning still yields long‑term balance because each batch randomly selects a new partition; over many batches the distribution evens out. In the short term, it concentrates writes, which aligns with broker‑level sequential disk writes and is friendlier to the broker.

Limitations:

Only applies to messages with a null key; keyed messages continue to use hash.

Latency‑sensitive workloads may suffer because messages wait for the batch to fill.

If a selected partition is unhealthy, an entire batch may be sent to it, potentially causing larger loss than random routing.

Therefore sticky partitioning is usually combined with the next‑generation “intelligent” strategies.

Fourth Generation Strategy: Intelligent Partitioning

At 10 M QPS, partition selection becomes an online control problem. Intelligent partitioners adjust the probability of selecting each partition based on runtime feedback.

Load‑aware : producers periodically collect per‑partition write latency, queue time, and compressed size, compute a health score, and perform weighted random selection favoring healthy partitions. Kafka’s evolution from UniformStickyPartitioner to BuiltInPartitioner follows this path.

Latency‑aware : in multi‑region deployments, network RTT to each partition varies. The selector prefers partitions with lower RTT, spilling traffic to higher‑RTT partitions only when the low‑RTT ones are overloaded. Typical use‑case: financial trading where tail latency matters.

Lag‑aware : the producer observes each partition’s consumer lag; if a partition is heavily lagging, new messages are routed elsewhere. RocketMQ has begun exploring similar designs.

All three variants share the same core idea: turn a static bucket into a feedback‑driven load‑balancer.

The price is added complexity: sampling frequency, aggregation, smoothing, and avoiding oscillation. Too‑frequent sampling stresses the broker; too‑sparse sampling reacts slowly. Aggressive weighting can cause avalanche effects; conservative weighting may leave the system unchanged.

Hot‑Key and Partition Skew Mitigation

Even with intelligent partitioning, hot keys remain because the strategy only chooses healthier partitions, not lower‑traffic keys. Two engineering approaches are described:

Key suffix splitting : append a sub‑suffix to a hot key, e.g., user_12345 → user_12345_0, user_12345_1, … user_12345_N, distributing the same logical user across N partitions. This sacrifices strict per‑key ordering but enables horizontal scaling for “locally ordered” workloads.

Two‑level routing : first select a coarse‑grained partition (e.g., by tenant ID), then within that partition apply key‑hash to choose a physical partition. Hot keys affect only a subset of partitions belonging to the same tenant, preventing system‑wide hot‑spot spread. Similar designs exist in Kafka Streams’ repartition and Pulsar’s key_shared mode.

In large‑scale systems, a single strategy is rarely sufficient; teams build a “strategy map” that assigns different strategies per topic, per business domain, and per traffic pattern.

Engineering Actions for Production

Four concrete steps are recommended:

Choose the default strategy : if no ordering is required, use sticky partitioning; if ordering is needed, profile key distribution. For hot‑key ratio < 1 % and 16‑64 partitions, use key hash; for higher ratios or more partitions, consider two‑level routing or suffix splitting.

Shift load‑testing focus : beyond QPS and throughput, inject uneven key distributions and fault scenarios, and observe p99/p999 latency under hotspot and partial failure conditions.

Close the monitoring loop : maintain three charts—partition write‑rate distribution, consumer lag distribution, and leader placement. Examine p99 and extreme values, not just averages.

Make the strategy configurable : abstract the partitioner into a hot‑swappable component. Different topics can use different strategies, and the same topic can switch between strategies for peak vs. off‑peak periods.

From Function to System Perspective

Returning to the 2 am alarm, the team applied three actions: switched the default to sticky partitioning, introduced suffix splitting for hot keys, and added partition‑rate p99 as a core SLO. Two years later the issue never resurfaced.

The evolution of partition selection is a perspective shift: from a trivial “place‑message‑somewhere” function to a system‑level design that intertwines throughput, ordering, hotspot handling, balance, and fault isolation. Each generation—random, sticky, key‑hash, intelligent—moves the concern from a client‑side detail to a central architectural decision.

For anyone building or optimizing a high‑throughput message‑queue system, ask yourself: Is your partition selection explicitly designed or inherited from defaults? Have you profiled key distribution? Does your load test cover hotspots and failures? Is your strategy driven by configuration or hard‑coded?