Kafka Deployment Mastery: ZooKeeper vs KRaft Modes + Full Automation Scripts

Kafka Core Concepts

Apache Kafka is a distributed streaming platform originally developed by LinkedIn and open-sourced in 2011. It features high throughput, low latency, and horizontal scalability. Key concepts are summarized in a table with analogies:

Producer – sends messages (courier sender)

Consumer – reads messages (courier receiver)

Broker – single Kafka server node (courier outlet)

Topic – logical message category (courier route)

Partition – physical partition of a topic, ordered immutable sequence (sorting lane)

Offset – unique message identifier in partition (tracking number)

Consumer Group – group of consumers sharing a topic for load balancing (team picking up packages)

ZooKeeper – coordination service for metadata and leader election (pre-2.8) (central dispatch center)

Deployment Preparation

Kafka requires two hard dependencies:

JDK – Kafka is written in Java; JDK 8 is recommended for stability.

ZooKeeper – required for Kafka versions before 2.8; still recommended for beginners to simplify troubleshooting.

System requirements: CentOS 7.x/RHEL 7.x (or Ubuntu with apt), minimum 2 GB RAM (4 GB+ recommended), 10 GB+ disk space for logs.

An environment check script ( check-env.sh) is provided for Linux and macOS. It verifies OS, memory, Java version (1.8/11/17), key ports (2181 ZooKeeper, 9092 Kafka, 9093 SSL), disk space, and Homebrew on macOS. Output shows PASS/WARN/FAIL with installation hints.

Option 1: ZooKeeper + Kafka Single-Node Deployment

Traditional, stable approach with clear separation of concerns; easiest for beginners to debug.

ZooKeeper Deployment

Download :

wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz

(multiple mirrors listed).

Extract & symlink : tar -zxvf ... -C /usr/local/ and

ln -s /usr/local/apache-zookeeper-3.8.4-bin /usr/local/zookeeper

(macOS uses ~/Applications).

Configure data directory : create /usr/local/zookeeper/data, copy zoo_sample.cfg to zoo.cfg, change dataDir from /tmp/zookeeper to the new path using sed.

Start & verify : /usr/local/zookeeper/bin/zkServer.sh start then status; Mode: standalone indicates success.

Kafka Deployment

Download :

wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.6.1/kafka_2.13-3.6.1.tgz

(verify ~900 MB). macOS uses curl -L -# -O.

Extract & symlink : tar -zxvf ... -C /usr/local/, ln -s /usr/local/kafka_2.13-3.6.1 /usr/local/kafka, create /usr/local/kafka/logs.

Core configuration ( server.properties):

broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://$(hostname -I | awk '{print $1}'):9092
log.dirs=/usr/local/kafka/logs
zookeeper.connect=localhost:2181/kafka
log.retention.hours=168
log.segment.bytes=1073741824

macOS uses variables USER_NAME and KAFKA_IP for paths and advertised listener.

Start :

/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

, wait 5 seconds, verify with jps | grep Kafka.

Functional Verification

Create test topic, produce and consume a message:

/usr/local/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
echo "测试Kafka连通性，节后直接用" | /usr/local/kafka/bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
/usr/local/kafka/bin/kafka-console-consumer.sh --topic test --bootstrap-server localhost:9092 --from-beginning --max-messages 1

Option 2: KRaft Mode Deployment (No ZooKeeper)

Introduced in Kafka 2.8.0, KRaft embeds the coordination controller, removing ZooKeeper dependency. Simpler deployment, fewer failure points, recommended for production.

Generate cluster ID : KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid).

Format storage :

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

.

Modify config/kraft/server.properties (using sed): enable process.roles=broker,controller, set node.id=1, controller.quorum.voters=1@localhost:9093, listeners for PLAINTEXT (9092) and CONTROLLER (9093), advertised listener with host IP, log directory, and cluster ID.

Start & verify :

bin/kafka-server-start.sh -daemon config/kraft/server.properties

, wait, jps | grep Kafka (no ZooKeeper process). Test same as Option 1.

KRaft Automation Script

A comprehensive Bash script ( kafka-kraft-deploy.sh) automates the entire KRaft deployment on Linux and macOS. Key features:

Parameterized : install directory, local package or auto-download, Kafka version (default 3.6.0), ports (9092/9093), log directory, run user, force reinstall, auto-confirm.

OS detection : identifies Linux distribution (Ubuntu, Debian, CentOS, RHEL, Rocky, AlmaLinux) and macOS.

Dependency checks : Java 11+, curl, tar; auto-install via apt/yum/brew if missing and confirmed.

Port conflict detection using ss (Linux) or lsof (macOS).

Linux user creation (e.g., kafka system user).

Installation methods : local .tgz/.tar.gz package or auto-download from Apache mirrors (official, archive, Tsinghua).

KRaft config generation : creates config/kraft/server.properties with tuned defaults (G1GC, heap 1G, network/IO threads, log retention, replication factors, metrics reporter placeholder). Also creates kafka.env with JVM opts and paths.

Storage formatting : generates cluster ID, creates log directory, runs kafka-storage.sh format.

Management scripts : kafka-start.sh, kafka-stop.sh (graceful then force kill), kafka-status.sh (health check, topic list, recent logs), kafka-test.sh (create topic, produce/consume, cleanup).

Systemd service (Linux): creates /etc/systemd/system/kafka-kraft.service with restart policy, file limits, environment file.

Startup & test : waits up to 30 seconds for port listening, runs quick test if confirmed.

Summary output : install paths, ports, user, cluster ID, management commands, systemd commands, next steps.

The author advises testing the script in a non-production environment first, as edge cases may require debugging.