Queue Selection at Ten‑Million QPS: From General‑Purpose to Specialized Queues

1. Midnight alert

Late on a Friday night an alarm reported 4.2 million pending order‑confirmation emails and an 80‑fold increase in complaints within five minutes. Core transaction latency was normal, but a Kafka topic’s consumer latency jumped from seconds to hours while the producer continued at ~120 k TPS. The downstream consumer cluster handling email notifications had only 32 partitions; scaling the consumer group to 64 instances left half idle, and adding partitions required a rebalance that stalled the group for minutes. This raised the question: why use Kafka for this task?

2. Million‑QPS era: a “Swiss‑army knife”

Typical architecture at this scale: Nginx → Java/Go services → MySQL + Redis → a single Kafka cluster for all asynchronous workloads (orders, logs, click‑streams, callbacks, notifications). The generic‑queue approach minimized operational overhead—one cluster, one monitoring setup, familiar tooling—but a single‑purpose solution cannot excel at every workload, and the model begins to break when scaling to ten‑million QPS.

3. Ten‑million QPS is a qualitative shift

3.1 Topic explosion

Topic count grows from dozens to thousands, stressing controller load, ZooKeeper watches, and broker file‑handle limits.

3.2 Scenario diversity

Order payment events: strict order, strong consistency, no loss.

User‑behavior logs: tolerate some loss, require extreme throughput, no ordering.

Delayed tasks: precise timing, cancellable, second‑level granularity.

IM messages: low latency, per‑session ordering, broadcast.

IoT device reports: massive concurrent connections, small‑packet high‑frequency, per‑device stickiness.

Using a single generic queue for all these workloads yields sub‑optimal results.

3.3 Failure impact expands

At million QPS a Kafka hiccup affects a batch of services; at ten‑million QPS a single overloaded cluster can cripple the entire company.

4. Five hidden limits of a generic queue

4.1 Partition count vs. concurrency

Each partition caps consumer concurrency. Scaling from 32 to ~2 000 partitions is needed for ten‑million QPS, but metadata overhead and rebalance latency grow linearly.

4.2 Ordering vs. parallelism

Kafka guarantees order only within a partition. Enforcing per‑user ordering by hashing keys creates hot partitions, sacrificing parallelism.

4.3 Persistence model vs. latency

Sequential disk writes give high throughput but typical end‑to‑end latency is tens of milliseconds, unsuitable for IM or real‑time matching.

4.4 Consumption model vs. scheduling

Pull‑based consumption fits streaming but not scheduled tasks; implementing delayed execution requires ad‑hoc Redis ZSets, timers, and custom code.

4.5 Protocol stack vs. connectivity

Kafka’s server‑to‑server protocol works poorly for weak networks, mobile clients, or massive IoT devices that need MQTT or WebSocket.

5. Five typical specialized queue categories

5.1 Transactional queues

Products: RocketMQ transactional messages, Apache Pulsar

Features: half‑message or two‑phase commit, strict ordering, full ACK/DLQ, slightly lower throughput than Kafka

Use‑cases: orders, payments, accounting

5.2 Throughput‑optimized queues

Products: Kafka, Pulsar

Features: sequential writes, batch flushing, zero‑copy, high per‑broker TPS; ordering only within partitions

Use‑cases: user‑behavior logs, click‑streams, CDC, data‑lake ingestion

5.3 Delayed queues

Products: RocketMQ delayed messages, RabbitMQ TTL + DLX, Redis Stream + ZSet, custom timer wheels

Features: N‑second/minute delivery, message‑level TTL, cancellable, often built on time‑wheel + layered timers

Use‑cases: order cancellation, coupon expiry, scheduled pushes

5.4 Streaming queues

Products: Kafka + Kafka Streams, Pulsar + Flink, Pulsar Functions

Features: stateful processing, windows, exactly‑once semantics, data appears as stream and table simultaneously

Use‑cases: real‑time recommendation, fraud detection, streaming analytics

5.5 Device‑centric queues

Products: EMQX, HiveMQ, Alibaba Cloud MQTT, Tencent Cloud IoT Hub

Features: MQTT/WebSocket protocols, millions of concurrent connections, QoS levels, topic wildcards, retained messages

Use‑cases: IoT telemetry, IM push, mobile events, vehicle messaging

6. Three‑question decision framework

How much loss is acceptable?

Zero loss → transactional queue

Low loss tolerable → throughput queue

Loss acceptable → streaming or device queue

How latency‑sensitive is the message?

Millisecond‑level → device or delayed queue

Second‑level → throughput or transactional queue

Minute‑hour‑day level → delayed queue (avoid forcing Kafka to act as a timer)

What ordering guarantees are needed?

Global strict order → single‑partition transactional queue (watch throughput ceiling)

Per‑business‑key order → transactional or throughput queue with key hashing

No order needed → throughput queue

These questions resolve ~90 % of scenarios to a clear queue choice.

7. Real‑world comparison of five main queues

Kafka’s partition count is limited by controller and metadata memory.

Pulsar’s “million‑topic” capability stems from its stateless broker architecture.

RocketMQ excels at transactions and delayed messages, not raw throughput.

RabbitMQ has the lowest TPS but the richest routing (topic, fanout, header).

EMQX focuses on connection count, not TPS, making it incomparable to Kafka.

8. Evolution from a single queue to a queue matrix

Below 100 k QPS: a single Redis (list or stream) suffices; optionally a small Kafka backup.

100 k – 1 M QPS: one Kafka cluster handles everything – the “Swiss‑army knife” stage.

1 M – 10 M QPS: split into transactional and throughput streams; begin a layered queue architecture.

10 M QPS and above: deploy a queue matrix behind a message‑gateway, with dedicated SRE, monitoring, and capacity planning for each of the five specialized queues.

9. Core insight: scenario‑driven architecture

In the midnight‑alert case the email‑notification path was migrated from the main Kafka cluster to an independent RocketMQ delayed queue. Order events remain on Kafka; delayed email tasks run on the specialized queue, isolating failures and preserving the primary transaction flow.

Specialized queues are not added for complexity’s sake but to match each scenario with the most appropriate tool—just as a kitchen needs a chef’s knife, a scalpel, and scissors, a ten‑million‑QPS system needs transactional, throughput, delayed, streaming, and device‑oriented queues.