Monitoring Kafka Consumer Groups with kafka-consumer-groups and Kafka Manager

When dealing with slow MySQL queries, the author suggests precomputing results into MongoDB or Elasticsearch, requiring MySQL binlog data to be sent to Kafka and consumed in real time. Monitoring Kafka consumption is crucial because blockages can prevent up‑to‑date data from being available.

kafka-consumer-groups

The kafka-consumer-groups.sh script, located in Kafka's bin directory, lists all consumer groups and describes the consumption status of a specific group.

./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list

To view details of a group (e.g., Test-Group) run:

./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group Test-Group

The output includes fields such as TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID, HOST, and CLIENT-ID, which indicate the consumption lag and consumer status.

Kafka Manager

Kafka Manager provides a richer UI for monitoring consumer groups and managing topics and partitions. Download it from GitHub , unpack, and build with sbt clean dist. After configuring application.conf (e.g., setting kafka-manager.zkhosts) and running ./kafka-manager, access the web UI (default port 9000) to view consumer groups, broker partitions, and per‑topic consumption status.

Images in the original article illustrate the command output and the Kafka Manager UI.