How to Seamlessly Migrate from ActiveMQ to Kafka Without Service Disruption

Directly discuss the migration from ActiveMQ (AMQ) to Kafka, focusing on achieving a smooth transition that does not impact business operations or upstream/downstream systems.

Dependency Relationships

Before proposing a concrete solution, we outline the dependencies among systems. System H receives topics from upstream systems A and B and sends topics to downstream systems X, Y, and Z.

Message Types

Production (1‑1): one message from H consumed by a single downstream system (AMQ Queue).

Production (1‑N): one message from H consumed by multiple downstream systems (AMQ VirtualTopic).

Consumer type: messages sent by upstream A or B and consumed by H.

Self‑produce‑consume: H both produces and consumes the message.

VirtualTopic

Production (1‑N) messages should be treated as VirtualTopic scenarios rather than plain Topics. The diagram below shows the difference.

When system H sends a VirtualTopic named VirtualTopic.PAY_SUCCESS_ORDER, instances X‑1, X‑2, X‑3 of system X and Y‑1, Y‑2, Y‑3 of system Y each have a single consumer group:

VConsumers.memberGroup.VirtualTopic.PAY_SUCCESS_ORDER

VConsumers.pointIssue.VirtualTopic.PAY_SUCCESS_ORDER

If H sends a plain Topic PAY_SUCCESS_ORDER, all instances of X and Y will receive the message.

Consumer Type

Since H is a consumer, we cannot predict when upstream A/B will switch to Kafka or when their backlog in AMQ is fully consumed. Therefore H must retain the original AMQ consumption logic and add a Kafka consumer in parallel.

Production (1‑1)

This scenario corresponds to AMQ Queue usage. H is the producer, and the downstream consumer relationship is simple (1‑to‑1). To avoid breaking downstream consumers that may not yet support Kafka, we keep the AMQ producer while adding a Kafka producer behind a switch. Once all downstream consumers have migrated to Kafka, the switch can be flipped and the AMQ code removed.

Production (1‑N)

This scenario corresponds to AMQ VirtualTopic usage. H is the producer, but multiple downstream consumers exist. Similar to the 1‑1 case, we retain the AMQ producer, add a Kafka producer behind a switch, and remove the AMQ code after all downstream consumers have migrated.

Self‑Produce‑Consume

When both production and consumption occur within H, we can replace AMQ with Kafka directly if there is no instance‑level deployment lag. If there is a lag, the migration becomes more complex.

1‑1

For a self‑produce‑consume 1‑1 message (H sends a Queue and also consumes it), the migration steps are:

Before rollout : All instances use AMQ for both sending and receiving.

After instance H1 rollout : H1 uses Kafka for sending and receiving; H2 and H3 remain on AMQ.

After instance H2 rollout : H1 and H2 use Kafka; H3 stays on AMQ.

1‑N

For a self‑produce‑consume 1‑N message (H sends a Topic and also consumes it), the migration steps are:

Retain AMQ consumption and add Kafka consumption in a new version.

Add Kafka production, remove AMQ production, and perform a rolling rollout until all instances run Kafka.

Summary

By following the above design, even when hundreds of systems across departments depend on MQ middleware, each system can be migrated gradually without affecting others, achieving a truly smooth migration.