Why Message Queues Matter: Decoupling, Asynchrony, and Real‑World Pitfalls

Why Use Message Queues?

Decoupling

In a typical e‑commerce system the transaction service calls three downstream services (order, inventory, warehouse). If any of these services is unavailable, the transaction service fails, creating a tight coupling.

By publishing a message to an MQ, the transaction service only depends on the MQ. It can continue operating while downstream services are down; once they recover they consume the buffered messages, achieving weak coupling.

Similarly, when a source system A must push data to multiple downstream systems (B, C, D …), direct HTTP/RPC calls require code changes for every new consumer and add retry/timeout logic. Replacing the calls with a single MQ publish lets each consumer subscribe independently, eliminating code changes in the producer when consumers are added or removed.

Asynchrony

Without an MQ the transaction service makes three synchronous calls; if each call takes 1 s the total latency is 3 s. With an MQ the calls become asynchronous, reducing the response time to under 1 s and improving overall performance.

Traffic Shaping (Peak Cutting)

When a burst of 5 000 requests arrives in one second but the order service can only handle 100 QPS, 4 900 requests would fail. An MQ buffers the requests; the order service pulls messages at its own pace, preventing overload.

This "peak cutting and valley filling" pattern also protects databases. If a database can sustain ~1 000 writes / s, a sudden spike to 5 000 writes would crash it. By routing writes through an MQ and limiting consumption to 1 000 QPS, the load is smoothed and the database remains stable.

Issues After Introducing MQ

Reduced System Availability

Adding an MQ creates an additional critical component. The overall system availability becomes the product of the availability of the original services and the MQ, lowering the combined uptime.

Increased System Complexity

The reliability of the whole system now depends on the MQ. Developers must handle message loss, ordering, and duplicate consumption. Typical mitigations include MQ clustering, partitioning for ordered consumption, and idempotent processing on the consumer side.

Duplicate Consumption

Duplicate messages can arise from:

Producer generating the same message multiple times.

Offsets in Kafka or RocketMQ being rolled back.

Consumer failing to acknowledge a message.

Acknowledgment timeout.

Manual retry logic in the business system.

Solution: implement idempotent handling by storing a consumption record with a unique messageId index. Before processing, check the table; if the ID exists, skip the message.

Data Consistency Issues

When calls become asynchronous, local transactions can no longer guarantee strong consistency, leading to scenarios such as an order being created while inventory deduction fails (overselling).

Adopt eventual consistency with retry mechanisms:

Low‑volume messages: synchronous retries (3‑5 attempts) then log to a table.

High‑volume messages: write failures to a retry table and let a scheduled job (e.g., XXL‑Job) reprocess later.

Message Loss

Typical loss scenarios:

Producer fails to send due to network issues.

MQ server encounters disk persistence errors.

Offsets are rolled back, skipping messages.

Consumer acknowledges receipt before business processing completes and then crashes.

Solution: maintain a sending table with a status field. After a producer sends a message, insert a record marked “pending”. The consumer updates the status to “confirmed” after successful processing. A periodic job scans for records still pending after a configurable timeout (e.g., 5 minutes) and resends them.

Message Ordering

Order‑related workflows (placed → paid → completed → refunded) require strict ordering. If a “paid” message arrives before a “placed” message, the system may process an invalid state.

When ordering is required, route all messages of the same order ID to the same partition (Kafka) or queue, ensuring sequential consumption. If ordering is not essential, focus on the final state and process messages in any order.

Message Backlog

If consumers cannot keep up with producers, messages accumulate in the MQ, delaying downstream actions (e.g., membership activation after an order).

Mitigation strategies:

If ordering is not required, consume messages with a thread pool; tune core and max pool sizes to match processing capacity.

If ordering is required, dispatch messages to single‑threaded logical queues after consumption.