Kafka MirrorMaker Mastery: Real‑Time Sync, Tuning & Troubleshooting

What is Kafka MirrorMaker

Kakfa MirrorMaker is the official Kafka solution for cross‑data‑center stream replication. It works by consuming messages from a source cluster and producing them to a target cluster using ordinary consumer and producer APIs, enabling near‑real‑time data sync with simple configuration.

Basic Features

When a target topic does not exist, MirrorMaker automatically creates an identical topic (same name, partition count, replica count) on the target cluster.

Multiple topics can be specified using a pipe‑separated list (e.g., TopicA|TopicB|TopicC) or commas.

Multi‑threaded consumption: each thread creates its own Consumer; adding more threads improves throughput if resources allow.

Horizontal scaling across processes is possible as long as all processes share the same consumer group. Partition assignment is handled automatically by the ConsumerGroup mechanism.

Consumer API Differences

Since Kafka 0.9 a new consumer API was introduced. Compared with the old high‑level/low‑level APIs, the new API:

Unifies the old APIs.

Removes the dependency on ZooKeeper for group management.

Is implemented entirely in Java, eliminating the need for a Scala runtime.

Provides improved security features.

MirrorMaker supports both old and new consumer APIs. By default it uses the old API; adding the --new.consumer flag switches to the new consumer.

Partition Assignment Strategies

Kafka offers two built‑in assignors: RangeAssignor (default) and RoundRobinAssignor . In MirrorMaker scenarios, RoundRobinAssignor yields a more balanced distribution.

Experiments with 6 topics (each with 2 partitions) and 80 consumer threads showed that RangeAssignor concentrates all partitions on a few consumers, while RoundRobinAssignor spreads them evenly, reducing lag and improving parallelism.

Performance Tuning

Thread Count : The --num.streams option controls the number of consumer threads. In the example, 40 threads per MirrorMaker instance were used (total 80 across two instances).

max.poll.records : Controls the maximum number of records returned in a single poll. Setting it too low reduces throughput; setting it too high can trigger frequent rebalances because heartbeats are sent only during poll calls.

Rebalance Avoidance : To minimize rebalances, keep max.poll.records moderate, and tune heartbeat.interval.ms, session.timeout.ms, and request.timeout.ms according to network stability.

Network Bandwidth : Throughput is limited by network bandwidth. Tests showed that a physical machine achieved 600 KB–1.5 MB/s, while virtual machines on the same host dropped below 100 KB/s.

Configuration Files

The consumer and producer properties are stored in mirror-consumer.properties and mirror-producer.properties. Example snippets are shown below:

Command‑Line Example

Starting MirrorMaker with the new consumer and a whitelist of topics:

nohup ./bin/kafka-mirror-maker.sh --new.consumer \
  --consumer.config config/mirror-consumer.properties \
  --num.streams 40 \
  --producer.config config/mirror-producer.properties \
  --whitelist 'ABTestMsg|AppColdStartMsg|BackPayMsg|WebMsg|GoldOpenMsg|BoCaiMsg' &

Monitoring Lag

Consumer group lag can be inspected with kafka-consumer-groups.sh. When using the new consumer, the --new-consumer flag must be added; otherwise the tool looks for offsets in ZooKeeper and reports no group information.

bin/kafka-consumer-groups.sh \
  --bootstrap-server ip-188-33-33-31.eu-central-1.compute.internal:9092,ip-188-33-33-32.eu-central-1.compute.internal:9092,ip-188-33-33-33.eu-central-1.compute.internal:9092 \
  --describe \
  --group africaBetMirrorGroupTest \
  --new-consumer

Conclusion

Kafka MirrorMaker is a reliable cross‑data‑center replication tool that must avoid data loss and keep replication speed above production rates. Proper load testing, careful tuning of thread count, poll size, partition assignor, and network‑related timeouts are essential to achieve optimal performance.