Why Use Message Queues? Scenarios, Pros, Cons, and HA Strategies

1. Why Use Message Queues? (Application Scenarios)

The interview question assesses why a message queue is needed and what problems it solves in a project.

Essence of a Message Queue

FIFO (first‑in‑first‑out) data structure.

Common scenarios: decoupling, asynchronous processing, traffic shaping (peak shaving).

Decoupling

Order systems often depend on payment, inventory, and logistics services. Using a queue separates these services, reducing tight coupling.

Asynchronous Processing

Traffic Shaping (Peak Cutting)

2. Comparison of Message Queue Products

MQ Selection Summary

ActiveMQ : Early adoption, not proven for large‑scale throughput, community less active, not recommended.

RabbitMQ : Implemented in Erlang, higher learning curve for Java developers, open‑source with active community, recommended for performance and stability.

RocketMQ : Java‑based, proven at Alibaba under high concurrency, stable and performant, recommended when custom development is needed.

Kafka : Suited for real‑time computing and log collection in big‑data scenarios, active community, recommended for big‑data and logging workloads.

3. Advantages and Disadvantages of Message Queues

Advantages : Decoupling, asynchronous processing, traffic shaping.

Disadvantages : Reduced system availability, increased complexity, consistency challenges.

4. Ensuring High Availability of Message Queues

RabbitMQ High Availability

Normal Cluster Mode

Messages are stored on a single broker instance; other nodes retrieve data via queue metadata. This does not provide true HA and introduces data‑pull overhead and a single‑instance bottleneck.

Mirror Cluster Mode

RocketMQ High Availability – Dual‑Master Dual‑Slave

5. Preventing Message Loss

Causes of Message Loss

Producer fails to send the message to the broker.

Broker crashes after receiving the message, causing in‑memory data loss.

Consumer encounters an exception before completing processing.

Loss‑Prevention Solutions

Reliable sending by the producer.

Persist messages on the broker.

Consumer acknowledges (ACK) only after successful processing; broker deletes the message after receiving ACK.

6. Avoiding Duplicate Consumption (Idempotency)

Root cause of duplicate messages: network unreachability.

Duplicate Sending Scenarios

During sending

During consumption

Idempotent Consumption Solution

Producer attaches a globally unique message ID.

Consumer checks DB/Redis for the ID before processing.

If not processed, consume and then record the ID.

If already processed, skip the message.

7. Ensuring Message Ordering

Global ordering : One producer, one queue, one consumer (1:1:1).

Partial Ordering

Producer sends messages of the same group to a specific queue.

Multiple consumers fetch from that queue concurrently.

The MQ uses segment locks to guarantee ordered consumption within the queue.

8. Handling Massive Message Backlog

Causes of Backlog

Network failures.

Consumers do not send proper acknowledgments after processing.

Backlog Mitigation Strategies

Identify and fix consumer processing speed.

Offload accumulated messages to a larger MQ cluster.

Scale out by adding more consumer nodes for parallel processing.

After consumption, revert to the original architecture.

9. Managing Expired Messages

Reasons for Expiration

Messages are given a TTL; if not consumed before the timeout, they are considered expired and can be moved to a dead‑letter queue.

Expiration Handling Process

Expired messages are routed to a dead‑letter queue.

A dedicated consumer processes the dead‑letter queue and logs entries to a database.

Operators can query the logs and resend messages to the main queue as needed.

That concludes the sharing for today.