Message Ordering at 10M+ QPS: From Global to Partition to Business-Key

Message ordering is not a binary switch but a spectrum that evolves from strict global ordering to partition‑level ordering and finally to ordering by business key. This article examines the cost of each level, common pitfalls, and practical approaches for achieving order at tens of millions of QPS.

Incident that triggered the discussion

In the early hours, an operations alert reported a user complaint: the order status showed "shipped" while payment had not been deducted. The DBA compared binlogs and found that for hundreds of orders the order status change flow appeared in reverse order: shipping → payment success. The root cause was a recent optimization: to push the payment topic throughput to 50 k QPS, the partition count was increased from 8 to 64 and the producer config max.in.flight.requests.per.connection was raised from 1 to 5. The change caused the two messages of the same order to be routed to different partitions, and producer retries further scrambled the intra‑partition order.

An architect summed up: "Message queue ordering is never a simple on/off switch; it is a continuous engineering commitment."

Why ordering matters

Teams often ignore ordering until anomalies such as "pay before order", "consume before recharge", or "replica newer than primary" appear. Scenarios that depend on state (e.g., order state machines, account balance updates) require strict ordering, while stateless or eventually consistent scenarios can tolerate disorder.

First step: identify which messages truly need ordering before choosing an implementation.

Global ordering – the naive single‑writer model

Global ordering guarantees that all messages are queued by production time and consumed in the same sequence. It is essentially a single writer + single reader + single partition design.

The downside appears immediately: the system’s throughput collapses to a single‑thread level. Benchmarks show that around 100 k QPS is the upper limit for a global‑ordered setup; beyond 1 M QPS it becomes untenable.

Hidden side effects include:

Single‑point failure amplification: if the broker hosting the sole partition glitches, the entire workflow stalls.

Slow consumer bottleneck: the slowest consumer drags the whole pipeline.

Retry logic dilemma: a failed batch may be retried after later batches succeed, breaking intra‑partition order.

Consequently, as traffic grows from 100 k to 1 M QPS, global ordering is usually replaced by a more flexible partition ordering approach.

Partition ordering – ordering per business key

Most businesses only need ordering for messages that share the same business object. Examples:

Order status changes must be ordered per order, but different orders can be processed in parallel.

User balance updates must be ordered per user, but different users are independent.

Device uplink/downlink messages must be ordered per device, but different devices can run concurrently.

The solution is to route messages with the same business key to the same channel (partition). Within a channel order is preserved; across channels processing is parallel, allowing horizontal scaling.

Major MQ products implement this model as the default ordering guarantee.

Three critical decisions affect partition ordering:

Business‑key selection

Partition count

Hot‑spot mitigation

Choosing the business key

The business key defines the minimal unit of ordering. Three principles guide the choice:

Cover state dependencies: all events of a state machine must share the same key (e.g., order ID for order state, account ID for balance, device ID for device sessions).

Avoid hotspots: a key that is too coarse (e.g., merchant ID when a few merchants generate 90 % of traffic) will concentrate load on a few partitions, effectively reverting to global ordering.

Deterministic generation: the same business event must always produce the same key. Using timestamp + order ID fails because retries change the timestamp, causing different partitions.

Real‑world mistakes:

An e‑commerce system used user ID as the key. A large customer placing hundreds of orders caused a single partition to overload.

An IoT platform used product model as the key. Millions of devices of the same model saturated a partition; switching to device ID resolved the issue.

A financial system used both account ID and merchant ID for the same transaction, causing related messages to land in different partitions and break reconciliation.

Choosing the right key is a balance between ordering granularity and parallelism.

Determining partition count

Once the business key is fixed, the next decision is how many partitions to allocate. Changing the partition count later is risky because the hash distribution changes, breaking the ordering guarantee for existing keys.

Typical estimation dimensions include traffic volume, key cardinality, and expected growth. An example illustrates the problem:

With 8 partitions, order ID 123456 hashes to partition 5, so both order and payment messages land in partition 5. Expanding to 16 partitions may rehash the same order ID to partition 13, while the previous day’s messages remain in partition 5. Consumers processing partitions concurrently could consume the new shipping message before the older payment message, corrupting the state machine.

Scaling strategies for partition count

Reserve enough partitions early: estimate 3‑5 years of peak traffic and provision hundreds to thousands of partitions per topic (common in Kafka clusters handling 10 M QPS).

Add a routing layer on the business side: use consistent‑hashing with virtual nodes to mask partition changes during scaling.

Pause writes for hot keys during migration: stop producing for affected keys, let the backlog drain, then switch partitions.

Parallel new topic: create a new topic with more partitions, gradually shift traffic, and retire the old topic after it is fully consumed.

The three gates of ordering

After the business key and partition count are set, ordering must survive three processing stages: production, transmission, and consumption.

Gate 1 – Production side must route the same key to the same partition

Consistent key extraction: the same business event must yield the identical key across services, versions, and machines. Embedding the extraction logic in an SDK helps enforce consistency.

Producer retries must not reorder: Kafka’s max.in.flight.requests.per.connection defaults to 5, allowing up to five un‑acknowledged batches. If a batch fails and is retried, later batches may succeed first, breaking order. Set the parameter to 1 or enable idempotent producers with sequence numbers.

Transactional messages: ensure that all messages within a transaction are committed in order; otherwise, a single order’s events could be split and arrive out of sequence.

Gate 2 – Transmission and storage must preserve FIFO within a partition

Most MQs guarantee FIFO per partition, but leader election and broker failover can break it. Kafka’s unclean.leader.election=true may promote a stale replica, violating order. Disable this flag in production.

During consumer group rebalance, partitions are reassigned. While at‑least‑once delivery usually prevents loss, concurrent processing by old and new consumers can cause temporary duplication or out‑of‑order consumption. Use cooperative (sticky) rebalance, static membership ( group.instance.id), and design consumers to be idempotent.

Gate 3 – Consumption side must process messages serially per partition

A common mistake is to spawn multiple threads to handle a single partition for higher throughput. This works for unordered streams but breaks ordering for ordered streams.

Correct approach: perform a second hash on the business key and dispatch to a fixed thread in a local thread pool. The same key always uses the same thread, preserving order while allowing parallelism across keys.

Three cautions:

If the thread pool crashes, in‑flight messages must be re‑fetched from the offset.

Commit offsets only after all threads have finished processing their messages; otherwise, a crash could lose uncommitted messages.

Bound the thread‑pool queue length to avoid unbounded memory growth.

Details that matter at 10 M QPS

Hot‑spot imbalance

Even with a well‑chosen key, hot spots appear (e.g., flash‑sale items, large merchants). Three mitigation patterns:

Secondary split of hot keys: augment the key with a slot number (0‑9) so that messages for the same product are spread across ten partitions. This sacrifices intra‑product ordering but boosts throughput.

Dedicated channel for hot keys: create a separate topic or partition set for the top‑N hot keys, with its own consumer group.

Dynamic partition adjustment: detect hot partitions at runtime and split them into more sub‑partitions; merge cold partitions. Requires MQ support for dynamic partitioning.

Cross‑partition “pseudo‑order”

Sometimes business logic demands that message A precede message B even though they belong to different keys and thus different partitions. Native MQ mechanisms cannot guarantee this.

Two engineering solutions:

Handle the dependency in the business layer by attaching a version number or logical clock to each message. Consumers compare versions and discard stale messages, effectively turning the problem into a “last‑writer‑wins” scenario.

Merge the two keys into a larger business object so that both messages share the same key (e.g., use order ID for both payment and shipping events).

Introduce an orchestration service that receives both messages and enforces the required order, acting as a small single‑writer for that specific flow.

Rebalance storms

In a 10 M QPS cluster, consumer instances can number in the hundreds or thousands. Deployments, scaling, or network glitches trigger frequent rebalances, risking order gaps.

Common mitigations:

Sticky / cooperative rebalance: keep partition ownership stable, moving only the necessary partitions.

Static membership: assign a fixed group.instance.id to each consumer so short‑lived heartbeat losses do not trigger a rebalance.

At‑least‑once vs. at‑most‑once trade‑off: if occasional duplicates are acceptable, choose at‑least‑once with idempotent processing; if occasional loss is tolerable, at‑most‑once can skip uncommitted messages during rebalance.

Multi‑datacenter deployment

Ten‑million‑QPS workloads often span multiple data centers. The same business key may be produced in different regions, reaching different brokers and being consumed by different consumer groups, breaking global order even if each region maintains partition order.

Root solution: unitization – route all messages of a given key to the same logical unit (e.g., a specific region). Within the unit, strict order is kept; cross‑unit interactions are handled asynchronously, at the cost of added routing complexity and inter‑unit latency.

Order vs. throughput trade‑off curve

Summarizing the discussion, the progression from 100 k QPS to 10 M QPS is essentially a gradual refinement of ordering granularity and a release of throughput capacity:

Global ordering → feasible up to ~100 k QPS.

Partition ordering → standard for ~1 M QPS; business‑key choice becomes decisive.

Business‑key ordering with fine‑grained routing → required for ~10 M QPS; must address hot spots, rebalance, and multi‑region challenges.

Early systems often over‑engineer ordering (e.g., choosing global ordering for safety). When traffic grows, the excessive ordering becomes a bottleneck, forcing a rushed downgrade that can cause cascading failures if the business side is unprepared.

Conclusion

Message ordering is an evolving commitment, not a binary choice. It moves along a spectrum:

global ordered → partition ordered → business‑key ordered

. Each step reduces the strictness of the guarantee but expands the throughput envelope, shifting responsibility from the platform to the business logic. Understanding this spectrum and the associated engineering trade‑offs is essential for building reliable, high‑throughput systems.