How to Diagnose and Resolve Message Queue Backlog Issues

Message Backlog Tracing

When a part of the system cannot keep up with upstream messages, a backlog builds up. A small backlog is normal, like water stored in a reservoir, but if the downstream processing capacity is insufficient, the water level (or message queue) keeps rising, indicating a problem.

Development Warnings

To prevent backlog, developers should optimize MQ usage. If a backlog already exists in production, the article outlines the best remediation steps.

Performance Optimization

The focus of performance tuning lies in the producer and consumer logic, not the MQ itself. Modern MQs can handle tens of thousands of messages per second per node and scale horizontally, while business logic typically processes only hundreds to a few thousand requests per second per node.

Producer Side

Producer performance is largely determined by the business logic that runs before sending a message. Ensure appropriate concurrency and batch sizes. A typical request‑response interaction with the broker takes about 1 ms; breaking this down:

Time spent preparing data, serializing the message, and constructing the request.

Network transmission time for the request and response.

Broker processing latency. 1000ms / 1ms * 1条/ms = 1000条消息 Sending one message per thread yields only about 1 000 messages per second, far below the MQ’s capacity. Increasing batch size or concurrency can multiply throughput. The choice depends on the nature of the producer:

If the producer is a microservice handling RPC requests, low latency is critical, so increasing concurrency is preferable.

If the producer is an offline analytics system that reads data from a database, batching is more suitable to maximize throughput.

In batch consumption, if any message fails, the entire batch is retried, effectively re‑sending all messages in the batch.

Consumer Side

Most performance issues appear on the consumer side. If consumption lags behind production, messages accumulate. A temporary lag can be tolerated, but a sustained imbalance leads to system instability, possible broker storage exhaustion, or message loss.

Therefore, the system must ensure that consumer throughput > producer throughput for long‑term stability.

Consumer optimization includes improving business logic, scaling horizontally, and increasing concurrency. When scaling consumer instances, the number of partitions (or queues) must be increased proportionally; otherwise, extra instances remain idle because each partition is consumed by a single thread.

Coordinating partition changes requires collaboration between the consumer team and the messaging middleware team.

A common but flawed approach is to decouple message receipt from business processing by immediately enqueuing messages into an in‑memory queue and returning. While this enables parallel processing, it risks message loss if the node crashes before the in‑memory queue is drained.

How to Handle Sudden Backlog Spikes

When a backlog suddenly grows, the root cause is usually either a surge in production speed or a slowdown in consumption. Most MQs provide built‑in monitoring to identify which side changed.

If production spikes (e.g., flash‑sale traffic), the quickest remedy is to scale out consumer instances. If scaling is not possible, degrade the system by disabling non‑essential services to reduce incoming traffic.

In Kafka, any change in the number of consumers or partitions triggers a rebalance.

If both production and consumption rates appear unchanged, investigate consumer failures that cause repeated retries, which can throttle the entire pipeline.

When consumption slows, examine logs for errors, and if none appear, capture stack traces to detect deadlocks or threads waiting on resources.

Some MQs offer a dead‑letter queue to isolate messages that repeatedly fail consumption due to format errors or other unrecoverable issues.

Summary

Key actions to address message backlog:

Producer optimization: increase batch size and/or concurrency.

Consumer optimization: improve business logic, scale out consumers, and ensure partition count matches consumer instance count.

Diagnose backlog using MQ monitoring metrics, log analysis, and stack traces.

Additional considerations:

Batch processing improves overall throughput but may increase latency and cause whole‑batch retries on failure.

For non‑critical messages (e.g., log backups), batch consumption is acceptable even if occasional loss is tolerable.

References:

《消息队列高手》

《RabbitMQ实战》

《Kafka实战》