Deploying and Using Kafka Monitor, Kafka Manager, and Kafka Eagle

This article explains how to deploy three Kafka management and monitoring tools: Kafka Monitor, Kafka Manager, and Kafka Eagle. It includes download links, required directory structures, configuration steps, startup scripts, and screenshots of the web interfaces.

1. Kafka Monitor

Download the Kafka‑Offset‑Monitor JAR from GitHub and place it in /opt/module/kafka-offset-console. Create a startup script start_en.sh with the following content:

#!/bin/bash
# 默认port为8086，由于我的环境8086端口已被占用，所有改为8087
java -Xms512M -Xmx512M -Xss1024K -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--offsetStorage kafka \
--kafkaBrokers cm1:9092,cm2:9092,cm3:9092 \
--kafkaSecurityProtocol PLAINTEXT \
--zk cm1:2181,cm2:2181,cm3:2181 \
--port 8087 \
--refresh 10.seconds \
--retain 2.days \
--dbName offsetapp_kafka 1>mobile-logs/stdout.log 2>mobile-logs/stderr.log &

Create the directory /opt/module/kafka-offset-console/mobile-logs, then run the script with ./start_en.sh. Access the monitor at http://cm1:8087 to view consumer/producer details.

2. Kafka Manager

Download the zip package from https://github.com/yahoo/kafka-manager/archive/1.3.3.15.zip, upload it to the cluster and unzip: $ unzip kafka-manager-1.3.3.15.zip Edit kafka-manager-1.3.3.21/conf/application.conf to enable basic authentication and set the Zookeeper hosts:

# 修改为：
kafka-manager.zkhosts="cm1:2181,cm2:2181,cm3:2181"
basicAuthentication.enabled=true
basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}
basicAuthentication.username="admin"
basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
basicAuthentication.password="admin"
basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
basicAuthentication.realm="Kafka-Manager"

Give execution permission and start the service on port 8090:

[root@cm1 bin]$ chmod 777 kafka-manager
[root@cm1 kafka-manager-1.3.3.15]$ nohup bin/kafka-manager -Dhttp.port=8090 > nohup.out 2>&1 &

Open http://cm1:8090 in a browser, log in with user admin, and you can add clusters, view brokers, topics, and perform administrative actions.

3. Kafka Eagle

Download the binary package from

https://github.com/smartloli/kafka-eagle-bin/archive/v1.4.8.tar.gz

and extract it twice:

cd /opt/module/
tar -zxf kafka-eagle-bin-1.4.8.tar.gz
cd kafka-eagle-bin-1.4.8
tar -zxf kafka-eagle-web-1.4.8-bin.tar.gz

Set environment variables:

export JAVA_HOME=/usr/java/jdk1.8.0_131-cloudera
export KE_HOME=/opt/module/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8
export PATH=$JAVA_HOME/bin:$KE_HOME/bin:$PATH

Modify system-config.properties to configure Zookeeper clusters and the web UI port (e.g., 8048):

kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=cm1:2181,cm2:2181,cm3:2181
kafka.eagle.webui.port=8048
kafka.eagle.driver=org.sqlite.JDBC
kafka.eagle.url=jdbc:sqlite:/opt/module/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8/db/ke.db
kafka.eagle.username=root
kafka.eagle.password=123456

Give execution permission to ke.sh and start the service:

chmod 777 ke.sh
./ke.sh start   # to stop: ./ke.sh stop

After a successful start, access the UI at http://<your_host>:8048. The interface provides cluster overviews, broker statistics, topic management, and a SQL‑like query feature.

All three tools together enable comprehensive monitoring, management, and troubleshooting of Kafka clusters.