Implementing Gray (Canary) Messaging for RabbitMQ and Kafka

Background: Existing RPC calls (Dubbo, Feign) support gray deployments, but MQ (RabbitMQ, Kafka) lacks this capability, leading to poor testing and production isolation. The article proposes a gray messaging solution for both RabbitMQ and Kafka.

Gray Scenarios: Two main approaches are discussed – (1) gray messages consumed only by gray nodes, and (2) gray messages can be consumed by normal nodes when gray nodes are absent, providing a fallback mechanism.

Gray Scheme Overview: The article evaluates a conventional shadow queue/topic approach and discards it due to exponential growth of queues/topics, operational overhead, and complexity, especially for RabbitMQ.

RabbitMQ Implementation

Uses the requeue feature. Producers add a gray flag to the message header, and consumers decide based on their own gray tag. If a normal consumer receives a gray message without a matching gray node, it requeues the message.

Production Process: At startup, setBeforePublishPostProcessors is registered on RabbitTemplate to inject the gray flag into the header before publishing.

Consumption Process: A custom listener overrides SimpleMessageListenerContainer.executeListener . The logic includes:

If the gray switch is off, process normally.

If the consumer matches the gray flag, consume directly.

Refresh gray instance cache via Eureka events; if no gray instance exists, consume normally.

If a gray instance exists and a normal node receives a gray message, handle prefetch limits and use a Bloom filter to avoid busy looping.

If none of the above apply, perform a Requeue.

Kafka Implementation

Leverages consumer groups. Gray messages are tagged in the header; gray consumers belong to a separate group ID. Normal groups perform a no‑op on gray messages, while gray groups process only messages that match the gray rule.

Production Process: A BeanPostProcessor adds an interceptor to KafkaTemplate to inject the gray flag into the header.

Consumption Process: When a node is identified as gray, its group.id is set to the gray group, and the interceptor executes the consumption logic.