Why Did My RocketMQ Consumer Stall? Uncovering the Docker Host‑Mode ClientId Bug

Preface

MQ users sometimes face message accumulation; the author recently hit this issue and discovered an unexpected cause.

Body

Late at night an alert "MQ message accumulation [TOPIC: XXX]" arrived, showing over 300 million piled‑up messages even though both producer and consumer servers reported normal traffic, disk I/O, and network conditions.

The author first suspected a RocketMQ bug because the open‑source version can handle billions of messages without performance loss.

Investigation of the consumer side showed that three consumer instances shared the same ClientId , which is unusual.

Producer speed >> Consumer speed Producer burst traffic. Consumer slowdown due to I/O block or crash.

Both producer and consumer applications appeared healthy, yet the backlog kept growing.

Problem Analysis

The identical ClientId was traced to Docker containers running in host network mode . In this mode, each container uses the host’s

docker0

bridge with the default IP

172.17.0.1

, so the client IP part of the ClientId is the same for all containers.

ClientId generation occurs in

ClientConfig.buildMQClientId()

:

public String buildMQClientId() {
  StringBuilder sb = new StringBuilder();
  sb.append(this.getClientIP());
  sb.append("@");
  sb.append(this.getInstanceName());
  if (!UtilAll.isBlank(this.unitName)) {
    sb.append("@");
    sb.append(this.unitName);
  }
  return sb.toString();
}

The IP is obtained via

RemotingUtil.getLocalAddress()

, which returns the address of

docker0

inside host‑mode containers.

Additionally, the instance name defaults to the process PID. Because all containers share the same PID namespace, the PID part is also identical, resulting in completely duplicated ClientIds.

Source Code Exploration

Key snippets:

private String clientIP = RemotingUtil.getLocalAddress();

public static String getLocalAddress() {
  // iterate network interfaces, prefer non‑loopback, non‑private IPv4
  // fallback to localhost
}

The load‑balancing logic in the consumer client uses the list of ClientIds (

cidAll

) to allocate message queues. When the list contains duplicate IDs, the index calculation yields the same result for each consumer, causing every consumer to be assigned the same queue.

int index = cidAll.indexOf(currentCID);
int mod = mqAll.size() % cidAll.size();
int averageSize = mqAll.size() <= cidAll.size() ? 1 :
    (mod > 0 && index < mod) ? mqAll.size() / cidAll.size() + 1 : mqAll.size() / cidAll.size();
int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
int range = Math.min(averageSize, mqAll.size() - startIndex);
for (int i = 0; i < range; i++) {
  result.add(mqAll.get((startIndex + i) % mqAll.size()));
}

Solution

Fix the load‑balancing error by ensuring each consumer has a unique ClientId. Set the environment variable

rocketmq.client.name

to a custom value, e.g., PID plus a timestamp:

@PostConstruct
public void init() {
  System.setProperty("rocketmq.client.name",
    String.valueOf(UtilAll.getPid()) + "@" + System.currentTimeMillis());
}

Address the message backlog after the ClientId fix; consumption speed returns to normal.

Summary

RocketMQ generates a consumer ClientId from the client IP and process ID.

In Docker host‑mode, all containers share the same

docker0

IP (172.17.0.1) and PID namespace, leading to identical ClientIds.

Consumer load‑balancing is performed on the client side; duplicate ClientIds cause every consumer to receive the same message queue, resulting in severe backlog.

Setting a unique

rocketmq.client.name

(or otherwise customizing the ClientId) resolves the duplication and restores normal consumption.