Message Queue Usage, Advantages, Disadvantages, and High‑Availability Design

Why Use Message Queues?

Interviewers often ask whether you understand why a system needs a message queue, what benefits it brings, and what drawbacks you must mitigate.

Decoupling : A producer sends data to a queue once; any number of consumers can read it later without the producer needing to know who consumes it.

Asynchronous processing : Time‑consuming downstream operations are off‑loaded to a queue, reducing request latency from hundreds of milliseconds to a few milliseconds.

Traffic shaping (peak‑shaving) : During traffic spikes, messages are buffered in the queue and consumed at a rate the downstream system can handle, preventing overload.

Message Queue Pros and Cons

The advantages are scenario‑specific (decoupling, async, peak‑shaving). The disadvantages include reduced system availability, added complexity (duplicate consumption, message loss, ordering), and consistency challenges.

Comparison of Popular MQs

Feature

ActiveMQ

RabbitMQ

RocketMQ

Kafka

Single‑node throughput

10⁴ msg/s (lower than RocketMQ/Kafka)

Same as ActiveMQ

10⁵ msg/s (high‑throughput)

10⁵ msg/s (high‑throughput, big‑data scenarios)

Topic count impact on throughput

—

—

Hundreds‑thousands of topics with modest impact

Throughput drops sharply when topics exceed a few hundred

Latency

ms‑level

µs‑level (lowest)

ms‑level

ms‑level

High availability

Master‑slave

Same as ActiveMQ

Distributed, very high

Distributed, very high (replication)

Message reliability

Low probability of loss

Almost none

Zero loss with proper config

Same as RocketMQ

Feature completeness

Very complete MQ features

Strong concurrency (Erlang), low latency

Complete, distributed, extensible

Simple MQ features, dominant in big‑data real‑time analytics

Recommendation: Small‑to‑medium companies often choose RabbitMQ for its stability; large enterprises with strong infra teams may prefer RocketMQ; big‑data pipelines typically adopt Kafka.

Ensuring High Availability

RabbitMQ HA Modes

Single‑node : Demo only.

Classic cluster (no HA) : Queues reside on a single node; other nodes only hold metadata.

Mirrored cluster (HA) : Queues are replicated to all nodes; a policy enables automatic mirroring.

Kafka HA Mechanism

Kafka uses a broker‑partition‑replica model. Each partition has a leader and one or more followers. With acks=all, replication.factor>1, and min.insync.replicas>1, writes are considered successful only after all replicas acknowledge, providing fault tolerance.

Idempotency and Duplicate Consumption

Both RabbitMQ and Kafka can deliver duplicate messages. To achieve idempotency, use unique identifiers, database primary‑key constraints, or Redis flags to detect and skip already‑processed messages.

Reliability (No Data Loss)

RabbitMQ: Enable message persistence ( deliveryMode=2) and publisher confirms ( channel.confirmSelect) to get ack / nack callbacks.

Kafka: Set acks=all, configure unlimited retries, and ensure sufficient replication.

Message Ordering

For strict order, keep a single partition per key and a single consumer thread, or route messages with the same key to the same in‑memory queue before parallel processing.

Handling Back‑pressure, Expiration, and Massive Accumulation

When queues fill up, temporarily scale out partitions/consumers, use a fast “drain‑only” consumer to empty the backlog, or discard and later re‑inject data after peak periods. TTL (TTL) can cause message loss; compensate by re‑publishing missing data.

Designing Your Own Message Queue

Key design points: distributed broker architecture (topic → partitions), sequential disk writes for high throughput, replication for HA, configurable retention, and support for idempotent consumption.

Choosing Between RPC and MQ

Use RPC (e.g., HTTP, gRPC, Dubbo) for synchronous request‑response interactions; use MQ for asynchronous, decoupled communication where latency tolerance is acceptable.

// Enable transaction in RabbitMQ
channel.txSelect
try {
    // send message
} catch (Exception e) {
    channel.txRollback
    // retry sending
}
// Commit transaction
channel.txCommit