Step-by-Step Guide to Deploying a Kafka Cluster and Essential Tools

Kafka Deployment Architecture

A Kafka cluster consists of multiple broker nodes that register their metadata in Zookeeper, forming a unified cluster.

PrettyZoo client

PrettyZoo is a JavaFX‑based graphical client built on Apache Curator for managing Zookeeper. It visualizes the Zookeeper nodes that store Kafka metadata.

Repository: https://github.com/vran-dev/PrettyZoo

Kafka Tool client

Kafka Tool provides a GUI for browsing topics, partitions and messages, making it easier to inspect cluster state.

Download page: https://www.kafkatool.com/

Kafka Eagle monitoring

Kafka Eagle visualizes the health of a Kafka cluster; it requires JMX to be enabled on each broker.

Project site: https://www.kafka-eagle.org/

Tip: Append a directory to the Zookeeper connection string (e.g., hadoop1:2181/kafka) to isolate Kafka metadata and avoid conflicts.

zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka

Zookeeper cluster deployment (3.4.6)

Upload the tarball and move it to /opt/apps/. mv zookeeper-3.4.6.tar.gz /opt/apps/ Extract the archive. tar -zxvf zookeeper-3.4.6.tar.gz Rename the sample configuration and edit conf/zoo.cfg.

mv conf/zoo_sample.cfg conf/zoo.cfg
# data directory
dataDir=/opt/apps/data/zkdata
# server list
server.1=hadoop1:2888:3888
server.2=hadoop2:2888:3888
server.3=hadoop3:2888:3888

Create the data directory and a myid file containing the node ID (1, 2, or 3).

mkdir -p /opt/apps/data/zkdata
echo 1 > /opt/apps/data/zkdata/myid   # on node 1
# repeat with 2 and 3 on the other nodes

Add Zookeeper to the system PATH.

export ZOOKEEPER_HOME=/opt/apps/zookeeper-3.4.6
export PATH=$PATH:$ZOOKEEPER_HOME/bin
source /etc/profile

Start, check, or stop the service.

bin/zkServer.sh start
bin/zkServer.sh status
bin/zkServer.sh stop

Kafka cluster deployment (2.11‑2.2.2)

Download Kafka 2.11‑2.2.2, move it to /opt/apps/, and extract.

mv kafka_2.11-2.2.2.tgz /opt/apps/
tar -zxvf kafka_2.11-2.2.2.tgz

Edit config/server.properties (example excerpt):

# Unique broker ID
broker.id=0
# Log directories
log.dirs=/opt/apps/data/kafkadata
# Zookeeper connection (note the /kafka suffix)
zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181/kafka
# Additional common settings
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Set Kafka environment variables.

export KAFKA_HOME=/opt/apps/kafka_2.11-2.2.2
export PATH=$PATH:$KAFKA_HOME/bin
source /etc/profile

Repeat the configuration on each node, changing broker.id accordingly.

Start or stop a broker.

# start
bin/kafka-server-start.sh -daemon /opt/apps/kafka_2.11-2.2.2/config/server.properties
# stop
bin/kafka-server-stop.sh

Kafka command‑line tools

Topic management

List all topics:

bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --list

Create a topic named first with 1 partition and replication factor 3:

bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --partitions 1 --replication-factor 3 --topic first

Describe a topic:

bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic first

Increase partitions (cannot decrease):

bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --alter --topic first --partitions 3

Delete a topic:

bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --delete --topic first

Producer

Send messages to a topic:

bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic first
> hello world
> xuyang hello

Consumer

Consume current messages:

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic first

Consume all messages from the beginning (including history):

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --from-beginning --topic first

Summary

The guide provides a complete step‑by‑step procedure for deploying a Zookeeper ensemble and a Kafka cluster, configuring the Zookeeper connection string with a dedicated /kafka node, and lists useful GUI tools (PrettyZoo, Kafka Tool, Kafka Eagle) for management and monitoring.