How to Set Up a Zookeeper & Kafka Cluster on CentOS 8 with Shell Scripts

1. Environment Preparation

Operating system: CentOS Linux 8.1.1911 (Core). Required software: JDK 8, Zookeeper 3.6.2, Kafka 2.7.0 built with Scala 2.13.

Download the three packages and place them in

/root/sf

(adjust the path as needed).

2. Install and Configure JDK

<code>tar zxvf jdk-8u221-linux-x64.tar.gz
# After extraction the directory is /root/sf/jdk1.8.0_221</code>

Append the following lines to

/etc/profile

(at the end of the file) to set

JAVA_HOME

and update

PATH

, then run:

<code>source /etc/profile</code>

2. Install and Configure Zookeeper

<code>tar zxvf apache-zookeeper-3.6.2-bin.tar.gz -C . /zk/   # extract to ./zk/</code>

Rename the extracted folder to the port number of each instance and create three instances:

<code>mv apache-zookeeper-3.6.2-bin 2181
cp -R 2181 2182
cp -R 2181 2183</code>

Create data directories:

<code>/root/sf/datas/zk/2181
/root/sf/datas/zk/2182
/root/sf/datas/zk/2183</code>

In each directory create a

myid

file containing 1, 2, and 3 respectively.

Copy the sample configuration and edit

zoo.cfg

(example shown in the image) to set

dataDir

,

clientPort

, and

server.x

entries that match the

myid

values.

Shell script to start all Zookeeper nodes

<code>#!/bin/bash
age="使用: $0 (start|stop|status)"
if [ $# == 0 ]; then
    echo $age
    exit 1
fi
behave=$1
echo "$behave zkServer cluster"
for i in 1 2 3
do
    /root/sf/zk/218$i/bin/zkServer.sh $behave
done
exit</code>

Make the script executable:

<code>chmod +x zkServer.sh</code>

Start the cluster with:

<code>./zkServer.sh start</code>

Verify with

jps

that the Zookeeper processes are running.

3. Install and Configure Kafka

<code>tar zxvf kafka_2.13-2.7.0.tgz -C . /kafka/   # extract to ./kafka/</code>

Rename the extracted folder to the broker port and duplicate it:

<code>mv kafka_2.13-2.7.0 9092
cp -R 9092 9093
cp -R 9092 9094</code>

Create data directories for each broker:

<code>/root/sf/datas/kafka/9092
/root/sf/datas/kafka/9093
/root/sf/datas/kafka/9094</code>

Edit each

server.properties

(example shown in the image) with the following key changes:

broker.id=0

(unique for each broker)

listeners=PLAINTEXT://:9092

(replace port accordingly)

log.dirs=/root/sf/datas/kafka/9092

(adjust for each broker)

zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

Shell script to start all Kafka brokers

<code>#!/bin/bash
age="使用: $0 (start|stop)"
if [ $# == 0 ]; then
    echo $age
    exit 1
fi
echo "$1 kafkaServer cluster"
for i in 2 3 4
do
    if [ "$1" == "start" ]; then
        /root/sf/kafka/909$i/bin/kafka-server-$1.sh -daemon /root/sf/kafka/909$i/config/server.properties
    fi
    if [ "$1" == "stop" ]; then
        /root/sf/kafka/909$i/bin/kafka-server-$1.sh stop
    fi
done
exit</code>

Make the script executable and start the cluster after Zookeeper is up:

<code>chmod +x kfServer.sh
./kfServer.sh start</code>

When both Zookeeper and Kafka show successful startup messages, the cluster is ready.