Why Your RocketMQ Consumer Stalls: Uncovering the Hidden VIP Port Issue

Preface

Hello everyone, good morning. After an overnight production system data‑center switch, I spent a sleepless night and decided to share my recent RocketMQ troubleshooting experience.

Problem Description

We launched a new business that listens to payment‑success MQ messages and pushes notifications to bound speakers. In the test environment everything worked, but after production release the speaker did not announce the successful payment because the MQ consumer failed to consume messages.

Initial checks showed no exceptions in the consumer logs. Manually publishing a message from the RocketMQ console succeeded, confirming the consumer could receive messages when sent directly.

The logs revealed the consumer repeatedly tried to connect to port 20878 and failed due to network issues.

RocketMQ broker starts three communication ports by default:

The configured broker port (default 10911 , we changed to 20880 ).

The VIP channel port, which is the first port minus 2 ( 20878 ).

The HA sync port, which is the first port plus 1 ( 20881 ).

Therefore the fix is either to open the VIP port in the firewall or disable the VIP channel.

Solution

Two ways to disable the VIP port:

Programmatically set the flag:

## Consumer
DefaultMQPushConsumer#setVipChannelEnabled(false)
## Producer
DefaultMQProducer#setVipChannelEnabled(false);

Set a JVM parameter:

-Dcom.rocketmq.sendMessageWithVIPChannel=false

Source Code Analysis

The root cause lies in the RebalanceService class, which performs topic rebalance. When the VIP port is unreachable, the consumer cannot fetch the full list of consumer IDs, leading to no queue assignment and thus no message consumption.

List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);

The broker’s VIP channel is enabled by default:

private boolean vipChannelEnabled = Boolean.parseBoolean(System.getProperty(SEND_MESSAGE_WITH_VIP_CHANNEL_PROPERTY, "true"));

Disabling it transfers all network requests to the regular broker port, avoiding the hidden VIP port issue.

Summary

The issue was caused by the VIP port 20878 being blocked, preventing the consumer from obtaining necessary metadata and thus from consuming messages. Opening the port or disabling the VIP channel resolves the problem. In production environments you should be aware of the four key ports:

9876 – NameServer listening port

10911 – Broker listening port

10909 – Broker VIP listening port

10912 – Broker HA sync port

When encountering mysterious RocketMQ problems, check network connectivity (e.g., telnet) and examine the broker logs located at ${user.home}/logs/rocketmqlogs.