Gray Deployment Solution for RocketMQ in the LUBAN MQ Platform

Background : RocketMQ is widely used as a message middleware for transaction management, asynchronous decoupling, peak shaving, and data synchronization. Existing gray‑deployment solutions for Dubbo/HTTP cannot fully address message isolation and switching, prompting the addition of a gray‑deployment extension to the LUBAN MQ platform.

RocketMQ Technical Characteristics

2.1 Storage Model : The commit log stores the actual message body. Topics are logical collections of messages; each topic is mapped to one or more queues (the physical execution units). Tags provide a second‑level classification within a topic, and consumer groups (identified by GroupID) isolate consumption streams.

2.2 Message Sending and Consumption : Producers obtain topic metadata, select a queue (default round‑robin or custom via MessageQueueSelector ), and send messages. Consumers pull messages based on their subscription (Topic, Tag) and maintain offsets. Offsets are updated to the smallest unconsumed offset to ensure correct replay after failures.

2.3 Subscription Consistency : All clients in the same consumer group must subscribe to identical Topic‑Tag sets; otherwise consumption logic becomes inconsistent and may cause message loss.

Industry Gray‑Deployment Schemes

Common approaches use separate consumer groups (different GroupIDs) or shadow topics, tags, or user properties to isolate gray traffic. These methods suffer from three main drawbacks: difficulty merging offsets after gray testing, ensuring complete consumption of gray messages, and managing the start point of gray offsets.

LUBAN MQ Platform Gray‑Deployment Solution

4.1 Queue Isolation : Designate configurable “gray queues” (e.g., the first and last queues of a topic) for gray messages while the remaining queues handle normal traffic. Both gray and normal consumers share the same GroupID, enabling seamless offset merging.

4.2 Broker Subscription Refactor : Add a graySubscriptionTable alongside the normal subscriptionTable . Heartbeat packets carry a grayFlag to indicate whether the client should use the gray or normal subscription set.

4.3 Producer Refactor : Implement MessageQueueSelector to route gray messages to the designated gray queues (configurable grayQueueSize ).

4.4 Consumer Refactor : Use a custom rebalance strategy ( AllocateMessageQueueGray ) that assigns gray queues to clients with grayFlag=true and normal queues to grayFlag=false . Before switching a gray client back to normal, the system checks that its gray queues are empty.

4.5 Namesrv Refactor : Store each client’s grayFlag in a lightweight, highly available Namesrv. The flag is persisted in a shared database to guarantee consistency across Namesrv instances.

Validation Scenarios

The document presents several demo scenarios (gray Topic/Tag unchanged, Topic added/removed, Tag changed, mixed changes) that verify the correctness of queue isolation, subscription handling, and offset merging. Each scenario includes subscription tables, queue allocation diagrams, and consumption results, confirming that gray traffic is isolated while normal traffic proceeds uninterrupted.

Conclusion

The LUBAN MQ gray‑deployment solution achieves full isolation of gray and normal messages using shared consumer groups, queue‑level segregation, and broker‑side subscription tables, while providing a reliable switch‑over mechanism via Namesrv‑managed flags. This approach eliminates the major drawbacks of existing industry solutions and enables loss‑less gray traffic migration.