Why Message Queues Matter: Solving Latency, Coupling, and Traffic Spikes

Preface

Message queues (MQ) have become increasingly popular in many companies, but why should you use them? This article answers three key questions: why use MQ, what new problems it introduces, and how to solve those problems.

1. Pain Points of the Traditional Model

1.1 Long Response Time

Complex business systems often require a single user request to synchronously call multiple downstream services, causing the overall response time to be long, especially under unstable network conditions, leading to timeouts.

The synchronous call pattern

Total response time is long

severely impacts user experience.

1.2 High Coupling

When a business is split into multiple subsystems (order, payment, inventory, points, logistics), a failure in any subsystem can cause the whole request to fail, indicating high coupling.

The subsystems are

too tightly coupled

, making the entire request fragile.

1.3 Traffic Spikes

Promotional activities (e.g., flash sales) can cause sudden traffic spikes that overwhelm the database, leading to slow responses or crashes.

During a spike,

request peaks

cannot be guaranteed to be stable.

2. Why Use MQ?

MQ can address the three problems above.

2.1 Asynchronous Processing

By converting synchronous calls to asynchronous messages, the system’s response time is significantly reduced.

The producer can return immediately while consumers process the business logic independently, avoiding

Total response time is long

.

2.2 Decoupling

Subsystems only depend on the MQ, eliminating strong inter‑service dependencies.

The order system produces messages and is unaffected by failures in payment, inventory, etc., reducing overall coupling.

2.3 Traffic Shaping (Peak Smoothing)

MQ buffers bursty requests, allowing consumers to process at their own pace, thus smoothing traffic spikes.

If a request peak occurs, messages stay in the queue until consumers can handle them, preserving system stability.

3. New Problems Introduced by MQ

3.1 Duplicate Messages

Duplicate consumption can happen due to producer duplication, offset rollback, consumer ack failures, timeouts, or manual retries.

Unhandled duplicates may cause data anomalies such as granting extra membership periods.

3.2 Data Consistency

If a consumer fails after the order is written but before points are awarded, the system ends up with inconsistent data.

Strong consistency would avoid this, but MQ typically provides eventual consistency.

3.3 Message Loss

Message loss can occur due to network failures on the producer side, disk errors on the broker, offset rollback, or consumer crashes after ack.

Any component (producer, broker, consumer) can cause loss, leading to data inconsistency.

3.4 Message Ordering

Stateful workflows (e.g., order → payment) require ordered processing; out‑of‑order messages can cause logical errors.

Different partitions or consumer threads may break ordering.

3.5 Message Backlog

If consumers process slower than producers, messages accumulate, delaying downstream actions such as membership activation.

Backlog leads to poor user experience.

3.6 Increased System Complexity

Introducing MQ adds another component (broker) to monitor and maintain, raising operational complexity and learning curve.

The architecture now includes producers, the MQ server, and consumers.

4. How to Solve These Problems

4.1 Duplicate Messages

Implement idempotent processing by using a consumption table with a unique

messageId

index; check the table before handling a message.

4.2 Data Consistency

Adopt retry mechanisms (synchronous for low‑volume, asynchronous for high‑volume) and use a retry table or job to reprocess failed messages, aiming for eventual consistency.

4.3 Message Loss

Maintain a message‑send table recording each produced message with a “pending” status; a periodic job checks for messages still pending after a timeout and republishes them.

4.4 Message Ordering

If strict ordering is required, route messages with the same key (e.g., order ID) to the same partition or queue, ensuring they are consumed sequentially.

4.5 Message Backlog

For unordered processing, use multithreaded consumers to increase throughput; for ordered processing, use single‑threaded consumers per partition or queue.