Why Use Message Queues (MQ) and How to Handle Common MQ Problems

Why Use MQ?

1.1 Decoupling

1.1.1 Decoupling 1

In an e‑commerce system the transaction service calls order, inventory, and warehouse services, creating strong coupling; if any service is unavailable the transaction fails.

Introducing an MQ lets the transaction service publish messages without caring about the downstream services, turning the coupling into a weak one and allowing the transaction to continue even if some services are temporarily down.

1.1.2 Decoupling 2

When many systems need the same data, adding direct HTTP/RPC calls quickly becomes unmanageable; using an MQ lets the producer publish once and consumers subscribe as needed, avoiding code changes for new consumers.

1.2 Asynchronous

Without MQ, a transaction service synchronously calls three services, taking about 3 seconds; with MQ the calls become asynchronous and the total latency drops below 1 second, greatly improving performance.

1.3 Traffic Shaping (Peak‑Shaving)

During spikes (e.g., 5 000 transactions per second) the order service may only handle 100 tps, causing failures; MQ buffers the requests so the order service can process them at its own pace.

1.3.1 Peak‑Shaving and Valley‑Filling

Databases often have a write capacity limit; MQ stores excess messages during peaks and releases them at a controlled rate, preventing overload and smoothing traffic.

Problems After Introducing MQ

2.1 Reduced System Availability

Adding MQ adds another component that must be kept alive, lowering overall system availability.

2.2 Increased System Complexity

Now the system relies on MQ for reliability, and issues such as message loss, ordering, and duplicate consumption must be addressed.

Solutions include clustering MQ for loss prevention, partitioning for ordering, and idempotent processing for duplicates.

2.3 Duplicate Consumption

2.3.1 Scenarios

Duplicate messages can arise from producer retries, offset rollbacks, consumer ack failures, timeouts, or explicit retries.

2.3.2 Solution

Implement idempotent handling, e.g., a consumption‑message table keyed by messageId to check whether a message has already been processed.

2.4 Data Consistency Issues

2.4.1 Scenario

Asynchronous processing breaks the ability to use local transactions, leading to possible inconsistencies such as overselling.

2.4.2 Solution

Adopt eventual consistency, use retry mechanisms (synchronous for low volume, asynchronous with a retry table or job for high volume), or employ MQ transactional messages where supported.

2.5 Message Loss

2.5.1 Scenarios

Loss can occur due to network failures on the producer side, disk errors on the broker, offset rollbacks, or consumer crashes after ack.

2.5.2 Solution

Maintain a send‑status table: producers write a pending record, consumers confirm receipt, and a scheduled job re‑sends messages still pending after a timeout.

2.6 Message Ordering

2.6.1 Scenarios

Stateful workflows (order → payment → completion) require correct ordering; mismatched order can cause logical errors.

2.6.2 Solution

Use partitioning by order ID so all related messages go to the same partition, or redesign to rely only on final state when ordering is not critical.

2.7 Message Backlog

2.7.1 Scenario

If consumer throughput is lower than producer rate, messages accumulate, delaying downstream processes such as membership activation.

2.7.2 Solution

For unordered workloads, increase parallelism with multithreaded consumers; for ordered workloads, route messages to single‑threaded queues after partitioning.