How to Set Up a Zookeeper & Kafka Cluster on CentOS 8 with Shell Scripts

1. Environment Preparation

Operating system: CentOS Linux 8.1.1911 (Core). Required software: JDK 8, Zookeeper 3.6.2, Kafka 2.7.0 built with Scala 2.13.

Download the three packages and place them in /root/sf (adjust the path as needed).

2. Install and Configure JDK

tar zxvf jdk-8u221-linux-x64.tar.gz
# After extraction the directory is /root/sf/jdk1.8.0_221

Append the following lines to /etc/profile (at the end of the file) to set JAVA_HOME and update PATH, then run:

source /etc/profile

2. Install and Configure Zookeeper

tar zxvf apache-zookeeper-3.6.2-bin.tar.gz -C . /zk/   # extract to ./zk/

Rename the extracted folder to the port number of each instance and create three instances:

mv apache-zookeeper-3.6.2-bin 2181
cp -R 2181 2182
cp -R 2181 2183

Create data directories:

/root/sf/datas/zk/2181
/root/sf/datas/zk/2182
/root/sf/datas/zk/2183

In each directory create a myid file containing 1, 2, and 3 respectively.

Copy the sample configuration and edit zoo.cfg (example shown in the image) to set dataDir, clientPort, and server.x entries that match the myid values.

Shell script to start all Zookeeper nodes

#!/bin/bash
age="使用: $0 (start|stop|status)"
if [ $# == 0 ]; then
    echo $age
    exit 1
fi
behave=$1
echo "$behave zkServer cluster"
for i in 1 2 3
do
    /root/sf/zk/218$i/bin/zkServer.sh $behave
done
exit

Make the script executable: chmod +x zkServer.sh Start the cluster with: ./zkServer.sh start Verify with jps that the Zookeeper processes are running.

3. Install and Configure Kafka

tar zxvf kafka_2.13-2.7.0.tgz -C . /kafka/   # extract to ./kafka/

Rename the extracted folder to the broker port and duplicate it:

mv kafka_2.13-2.7.0 9092
cp -R 9092 9093
cp -R 9092 9094

Create data directories for each broker:

/root/sf/datas/kafka/9092
/root/sf/datas/kafka/9093
/root/sf/datas/kafka/9094

Edit each server.properties (example shown in the image) with the following key changes: broker.id=0 (unique for each broker) listeners=PLAINTEXT://:9092 (replace port accordingly) log.dirs=/root/sf/datas/kafka/9092 (adjust for each broker)

zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

Shell script to start all Kafka brokers

#!/bin/bash
age="使用: $0 (start|stop)"
if [ $# == 0 ]; then
    echo $age
    exit 1
fi
echo "$1 kafkaServer cluster"
for i in 2 3 4
do
    if [ "$1" == "start" ]; then
        /root/sf/kafka/909$i/bin/kafka-server-$1.sh -daemon /root/sf/kafka/909$i/config/server.properties
    fi
    if [ "$1" == "stop" ]; then
        /root/sf/kafka/909$i/bin/kafka-server-$1.sh stop
    fi
done
exit

Make the script executable and start the cluster after Zookeeper is up:

chmod +x kfServer.sh
./kfServer.sh start

When both Zookeeper and Kafka show successful startup messages, the cluster is ready.