Mastering Message Queues: Key Questions Every Backend Engineer Should Know

Introduction

The author writes this article to help two friends, A and B, who work in traditional software and a state‑owned enterprise, respectively, to review the key points of message‑queue middleware.

Why write this article?

Friend A deals with product requirements, business logic, occasional SQL reports, and occasional bug fixes, experiencing no technical growth. Friend B works with middleware but only knows basic publish/subscribe without understanding why or how to ensure high availability.

Review Points

Why use a message queue?

What are the disadvantages of using a message queue?

How to select a message queue?

How to ensure high availability?

How to prevent duplicate consumption?

How to guarantee reliable transmission?

How to maintain message order?

The article provides a review framework rather than a full tutorial.

Main Content

1. Why use a message queue?

Three main scenarios: decoupling, asynchronous processing, and peak‑shaving.

Decoupling

Traditional mode tightly couples systems, requiring code changes when adding new systems. Middleware mode uses a queue so producers do not need to modify code.

In the middleware model, systems write messages to the queue and subscribe independently, eliminating direct dependencies.

Asynchronous Processing

Traditional synchronous execution wastes time. With a queue, non‑essential logic runs asynchronously, improving response speed.

Peak‑Shaving

During high concurrency, direct database access can cause connection failures. A queue buffers requests, allowing the system to consume at a sustainable rate.

2. Disadvantages of using a message queue

Reduced system availability: if the queue fails, dependent services may also fail.

Increased complexity: need to handle consistency, duplicate consumption, and reliable transmission.

3. How to select a message queue?

The author is familiar with ActiveMQ, RabbitMQ, RocketMQ, and Kafka. Community activity and release frequency are useful indicators.

Apache ActiveMQ 5.15.3 Release
... (records omitted)

ActiveMQ releases every few months, while RabbitMQ releases more frequently.

RabbitMQ 3.7.3 release 30 January 2018
... (records omitted)

For small‑to‑medium companies, RabbitMQ is recommended due to its high concurrency support and active community. Large companies may choose RocketMQ or Kafka based on data volume and specific use cases.

4. Ensuring high availability

High‑availability requires understanding cluster modes. For example, RocketMQ offers multi‑master and multi‑master‑slave configurations. Kafka uses ZooKeeper for leader election.

5. Preventing duplicate consumption

Duplicate consumption occurs when acknowledgments are lost. Solutions include using unique primary keys for database inserts, idempotent operations (e.g., Redis SET), or recording consumption IDs in a third‑party store.

6. Guaranteeing reliable transmission

RabbitMQ

Use transaction or confirm mode to avoid producer data loss, enable durable queues and persistent messages to prevent queue loss, and use manual acknowledgments to avoid consumer loss.

channel.addConfirmListener(new ConfirmListener() {
    @Override
    public void handleNack(long deliveryTag, boolean multiple) throws IOException {
        System.out.println("nack: deliveryTag = " + deliveryTag + " multiple: " + multiple);
    }
    @Override
    public void handleAck(long deliveryTag, boolean multiple) throws IOException {
        System.out.println("ack: deliveryTag = " + deliveryTag + " multiple: " + multiple);
    }
});

Kafka

Configure acks=all and retries=MAX on the producer, set replication.factor > 1 and min.insync.replicas > 1 to avoid broker loss, and use manual offset commits to prevent consumer loss.

7. Maintaining message order

Place ordered messages in the same queue/partition and consume with a single consumer, or handle ordering at the application level when multiple consumers are needed.

Conclusion

By mastering these questions, readers can cover most interview topics on message queues and demonstrate comprehensive understanding.