Why Message Queues Are Essential: Decoupling, Asynchrony, and Pitfalls Explained

1. Why Use a Message Queue?

1.1 Decoupling

1.1.1 Decoupling Example 1

For example, an e‑commerce system's core is the transaction service, which calls order, inventory, and warehouse services.

If any of those services is unavailable, the transaction service cannot run, creating a tight coupling.

Introducing an MQ lets the transaction service interact only with the MQ; it does not need to know whether the other services are up, achieving loose coupling.

Even if the other services are temporarily unavailable, the transaction service continues to run; once they recover, they consume the queued messages (degraded interface).

1.1.2 Decoupling Example 2

Suppose system A needs to push user data to systems B and C via HTTP or RPC. As more systems (D, E, …) need the data, A's code becomes tangled, and changes become painful. Using an MQ, A only publishes messages, and each consumer subscribes independently, eliminating code changes in A.

1.2 Asynchrony

Without an MQ, the transaction service synchronously calls three services, taking about 3 seconds if each call costs 1 second. With an MQ, calls become asynchronous and total latency drops below 1 second, greatly improving performance.

1.3 Traffic Shaping (Peak‑Shaving)

If 5 000 transactions arrive in one second but the order service can handle only 100 per second, 4 900 would fail. With an MQ, the transaction service enqueues messages and the order service processes them at its own pace, preventing overload.

1.3.1 Smoothing Peaks and Filling Valleys

Consider an order system that can write about 1 000 records per second to the database. During peak traffic, writes may surge to 5 000 + per second, which would crash the database.

Using an MQ, messages are stored in the queue and consumed at a controlled rate (e.g., 1 000 QPS), protecting the database from overload.

The term “peak‑shaving and valley‑filling” describes this: the MQ caps consumption speed, causing excess messages to accumulate (shaving the peak). After the burst, the consumer continues at the capped rate, gradually draining the backlog (filling the valley).

2. Problems After Introducing an MQ

2.1 Reduced System Availability

Adding an MQ adds another component that must remain available, lowering overall system availability.

2.2 Increased System Complexity

The transaction service no longer directly detects downstream failures; reliability now depends on the MQ.

We must address message loss, ordering, and duplicate consumption.

Solutions include MQ clustering for durability, partitioning for ordered consumption, and idempotent processing on the consumer side.

2.3 Duplicate Consumption

2.3.1 Scenarios

Duplicate consumption is common in MQs. Scenarios include producer‑generated duplicate messages, offset rollback in Kafka/RocketMQ, consumer acknowledgement failures, timeout during acknowledgement, and manual retries.

Unhandled duplicates can cause data anomalies such as granting extra membership time.

2.3.2 Solution

Implement idempotent handling, e.g., a consumption‑message table keyed by messageId. Before processing, check if the messageId has been handled; if so, skip processing.

2.4 Data Consistency (Asynchronous Distributed Transactions)

2.4.1 Scenario

Synchronous calls can use local transactions for strong consistency. With an MQ, calls become asynchronous, making strong consistency impossible and leading to issues such as overselling.

2.4.2 Solutions

Option 1: Keep critical operations (order, inventory) in a single transaction and use MQ only for non‑critical tasks.

Option 2: Use MQ transactional messages (available in RocketMQ). Transaction status can be Commit, Rollback, or Unknown.

CommitTransaction – the message is committed and can be consumed.

RollbackTransaction – the message is discarded.

Unknow – the broker will query the producer for the local transaction status.

Step 1: Producer Sends a Half Message

Before processing task A, the service sends a half message to the MQ.

The MQ persists the message without delivering it and returns an acknowledgement.

After receiving the ack, the service proceeds with task A.

When task A finishes, the service sends a Commit or Rollback request.

The MQ delivers the message to the consumer only after a Commit.

Step 2: MQ Delivers the Message to Consumer B

On normal flow, the MQ waits for the consumer’s ack and then finishes the transaction.

If the ack times out, the MQ retries delivery until the consumer confirms success; otherwise, manual intervention is required.

2.4.3 Consistency Model

MQs provide eventual consistency. To mitigate inconsistencies, add retry mechanisms—synchronous retries for low‑volume flows and asynchronous retries with a retry table and scheduled jobs for high‑volume flows.

2.5 Message Loss

2.5.1 Scenarios

Message loss can occur when the producer fails to send, the MQ crashes during persistence, offsets are rolled back, or a consumer acknowledges before processing and then crashes.

2.5.2 Solution

Maintain a message‑send table: after a producer sends a message, insert a record with status “pending”. The consumer updates the status to “confirmed” after processing. A periodic job checks for pending records and resends lost messages.

2.6 Message Ordering

2.6.1 Scenarios

Ordered processing is required for stateful data such as order lifecycle events. If a “pay” message arrives before a “create” message, the business logic breaks.

2.6.2 Solution

If strict ordering is not required, aggregate final state and process asynchronously. If ordering is required, route all messages of the same order to the same partition or queue, ensuring they are consumed sequentially.

2.7 Message Backlog

2.7.1 Scenario

When consumer throughput is lower than producer rate, messages accumulate, causing delays such as late membership activation after order placement.

2.7.2 Solution

If ordering is not needed, use multithreaded consumers to increase processing speed. If ordering is required, dispatch messages to multiple single‑threaded queues after consumption.