Message Queue Middleware: Why Use It, Drawbacks, Selection, High Availability, and Reliability

The author wrote this article to help two friends—one working in a traditional software outsourcing company and the other in a state‑owned enterprise—review the key points of message‑queue middleware.

Why use a message queue? The three main motivations are decoupling, asynchronous processing, and peak‑shaving. In a tightly coupled system, services call each other directly, making changes painful; a middleware architecture lets producers publish messages to a queue while consumers subscribe independently, reducing coupling and allowing asynchronous execution and controlled load during traffic spikes.

Drawbacks of using a message queue include reduced overall system availability (the queue becomes a single point of failure) and increased system complexity (additional concerns such as consistency, duplicate consumption, and reliable delivery must be handled).

How to select a message‑queue product —the author compares ActiveMQ, RabbitMQ, RocketMQ, and Kafka. Community activity, language support, throughput, latency, availability, and feature set are examined. For small‑to‑medium companies, RabbitMQ is recommended because of its Erlang‑based high concurrency and user‑friendly management UI; for large enterprises, RocketMQ or Kafka are suggested depending on data‑volume and use‑case requirements.

Ensuring high availability —different cluster modes are described. RocketMQ offers multi‑master, multi‑master‑slave asynchronous and synchronous replication. Kafka relies on Zookeeper for leader election and replication. RabbitMQ provides ordinary and mirrored clusters. Diagrams of typical cluster topologies are included.

Preventing duplicate consumption —the article explains idempotent design patterns: using a unique primary key for database inserts, leveraging Redis SET operations (which are naturally idempotent), or recording a global message ID in a third‑party store (e.g., Redis) before processing.

Reliable transmission —the three failure points (producer, broker, consumer) are analysed. For RabbitMQ, the author shows how to use transactions or the confirm mode; the confirm‑listener code is provided below:

channel.addConfirmListener(new ConfirmListener() {
    @Override
    public void handleNack(long deliveryTag, boolean multiple) throws IOException {
        System.out.println("nack: deliveryTag = " + deliveryTag + " multiple: " + multiple);
    }
    @Override
    public void handleAck(long deliveryTag, boolean multiple) throws IOException {
        System.out.println("ack: deliveryTag = " + deliveryTag + " multiple: " + multiple);
    }
});

Persistence is achieved by setting the queue to durable and publishing messages with deliveryMode=2. For Kafka, the author recommends producer config acks=all and retries=MAX, broker config replication.factor>1 and min.insync.replicas>1, and using manual offset commits to avoid loss caused by automatic commits.

Message ordering —to guarantee order, place all messages that must be processed sequentially into the same queue/partition (Kafka partition or RabbitMQ queue) and consume them with a single consumer. If higher throughput is needed, the author suggests handling ordering at the application level after consumption.

Conclusion —after studying these seven points, readers should be able to answer most interview questions about message queues, demonstrate a comprehensive understanding, and become thoughtful, capable programmers.