Why Did My RocketMQ Consumer Stumble? The Hidden ClientId Bug in Docker Host Mode

Preface

Users of message queues may encounter message accumulation. The author recently hit this problem and discovered an unexpected cause.

Story

One night a warning "MQ message accumulation [TOPIC: XXX]" arrived. After logging into the RocketMQ admin console (self‑hosted open‑source version) the author saw more than 300 million messages piled up.

The producer and consumer applications both appeared normal: server disk I/O, network, and traffic were all fine, yet the message backlog kept growing.

Producer speed >> Consumer speed Producer speed spikes suddenly. Consumer speed slows down, e.g., due to I/O blockage or crashes.

Since RocketMQ can support billions of queued messages, the huge backlog was clearly abnormal, suggesting a bug.

Problem Analysis

The author suspected the issue lay in the clientId generated by the consumer. In Docker host‑network mode, each container shares the host's

docker0

bridge, whose default IP is

172.17.0.1

. The clientId is built from the client IP and an instance name, so all consumers received the same IP and thus the same clientId.

ClientId generation occurs in

ClientConfig.buildMQClientId()

:

public String buildMQClientId() {
  StringBuilder sb = new StringBuilder();
  sb.append(this.getClientIP());
  sb.append("@");
  sb.append(this.getInstanceName());
  if (!UtilAll.isBlank(this.unitName)) {
    sb.append("@");
    sb.append(this.unitName);
  }
  return sb.toString();
}

The IP is obtained via

RemotingUtil.getLocalAddress()

, which iterates over network interfaces and prefers a non‑loopback, non‑private IPv4 address. In host mode, Docker containers all see the same

docker0

interface, so they all return

172.17.0.1

:

private String clientIP = RemotingUtil.getLocalAddress();

The instance name defaults to the system property

rocketmq.client.name

or "DEFAULT". If it remains "DEFAULT", the client changes it to the process PID:

private String instanceName = System.getProperty("rocketmq.client.name", "DEFAULT");

public void changeInstanceNameToPID() {
  if (this.instanceName.equals("DEFAULT")) {
    this.instanceName = String.valueOf(UtilAll.getPid());
  }
}

Because all containers share the same PID (Docker host mode gives each container the same process namespace), the final clientId becomes identical for every consumer.

Impact on Load Balancing

RocketMQ performs consumer load balancing on the client side. The broker stores each consumer's clientId in

consumerTable

. During rebalancing, the client fetches the list of message queues (

mqSet

) and the list of clientIds (

cidAll

), sorts both, and applies the default average‑allocation algorithm.

If

cidAll

contains duplicate clientIds, each consumer receives the same index (0) and therefore the same set of queues, causing all consumers to pull from the same queue and resulting in severe consumption slowdown.

private void rebalanceByTopic(final String topic, final boolean isOrder) {
  Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
  List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
  // ... sorting and allocation logic ...
  List<MessageQueue> allocateResult = strategy.allocate(
      consumerGroup,
      this.mQClientFactory.getClientId(),
      mqAll,
      cidAll);
  // ... update process queue table ...
}

Solution

The root cause is the identical clientId. To fix it, set a unique

rocketmq.client.name

environment variable for each consumer instance, e.g., combine the PID with a timestamp:

@PostConstruct
public void init() {
  System.setProperty("rocketmq.client.name",
      String.valueOf(UtilAll.getPid()) + "@" + System.currentTimeMillis());
}

After deploying the change, the message backlog began to shrink, confirming that the load‑balancing error was resolved.

Summary

RocketMQ consumer client generates a clientId as

IP@PID

(or custom name).

In Docker host‑network mode, all containers share the same

docker0

IP, leading to identical clientIds.

Consumer load balancing is performed on the client side; duplicate clientIds cause all consumers to receive the same message queues, resulting in slow consumption.

Setting a unique

rocketmq.client.name

(e.g., PID plus timestamp) breaks the duplication and restores normal consumption.