How to Diagnose and Resolve RocketMQ Message Accumulation Issues

1 Basic Concepts

When a consumer cannot keep up with the broker's sending speed, unprocessed messages accumulate, causing consumption delay.

Key scenarios that require attention to message accumulation and latency include mismatched upstream/downstream system capacities and strict real‑time consumption requirements.

2 Consumption Principle

The client uses Push mode and consumes messages in two stages:

Stage 1: Pull Messages – The client long‑polls the broker, batch‑pulls messages, and stores them in a local buffer. High throughput (tens of thousands of TPS per single‑threaded low‑spec machine) means this stage rarely becomes a bottleneck.

Stage 2: Consume Messages – Pulled messages are handed to consumer threads where business logic processes them. Consumption speed now depends on business processing time and concurrency; heavy or slow logic can fill the local buffer and stop further pulling.

Thus, the main bottlenecks are consumption time and consumption concurrency .

To avoid accumulation, prioritize reducing consumption time before increasing concurrency.

3 Consumption Bottlenecks

3.1 Consumption Time

Consumption time is affected by CPU/memory calculations and external I/O. Internal calculations are usually negligible compared to I/O such as database reads/writes, cache accesses, or downstream service calls (e.g., Dubbo, HTTP).

External calls must be profiled; unexpected latency spikes (e.g., a Dubbo call increasing from 20 ms to 200 ms during a promotion) cause backlog that cannot be solved merely by raising concurrency.

3.2 Consumption Concurrency

Most consumption is I/O‑bound; increasing parallelism can raise throughput, but beyond a point it degrades performance. Adjust concurrency by:

Adding more Consumer instances within the same ConsumerGroup (up to the number of subscription queues).

Increasing per‑instance thread pool via consumeThreadMin and consumeThreadMax parameters.

4 Resolution Strategies

When facing message accumulation, first identify the problematic stage.

4.1 Verify Consumption Time

Check whether consumption time is reasonable:

Print logs that record costTime for each message.

Inspect message trace visualizations.

Based on the measured latency:

If consumption time is long, examine the consumer JVM stack and optimize business logic.

If consumption time is normal, the issue may be insufficient concurrency; gradually increase consumer threads or scale out nodes.

4.2 Inspect Consumer JVM Stack

If consumption time is excessively high, retrieve the JVM stack:

Use jps -m or ps -ef | grep java to obtain the process ID.

Run jstack pid > stack.log to capture thread stacks.

Search for ConsumeMessageThread information, e.g., cat stack.log | grep ConsumeMessageThread -A 10.

Typical stack patterns:

Idle state: threads wait in WAITING for tasks.

Lock contention or sleep causing slow consumption.

Blocking on external I/O such as database or HTTP calls.

5 Summary

In Push mode, consumption consists of pulling messages and processing them. The primary bottlenecks are consumption time and concurrency. First analyze latency, then apply appropriate measures: optimize business logic when time is high, or increase concurrency when time is acceptable.

If consumption time is long, debug and optimize the consumer code.

If consumption time is normal but backlog persists, raise concurrency or add nodes.