Step-by-Step Guide to Installing and Running Kafka on Linux

1. What is Kafka

Kafka is a distributed, multi‑partition, multi‑replica messaging platform originally developed by LinkedIn and based on Zookeeper (newer versions no longer depend on Zookeeper). It is an open‑source publish‑subscribe message engine that offers high throughput, persistence, horizontal scalability, and stream processing capabilities.

Because Kafka provides high throughput, persistence, horizontal scalability, and stream processing, it is widely used.

2. Installing Kafka on Linux

(1) Download the installation package

Download URL: https://kafka.apache.org/downloads

Linux requires JDK 1.8 or higher.

(2) Extract the archive

# Extract files
tar -zxvf kafka_2.13-3.2.0.tgz
# Rename directory
mv kafka2.13 kafka

(3) Modify configuration files

cd /usr/local/kafka/config
vi server.properties
# broker instance identifier, must be unique in a cluster
broker.id=1
# directory for Kafka data
log.dirs=/tmp/kafka-logs
# Zookeeper connection string
zookeeper.connect=zk_ip:2181
# listener address
listeners=PLAINTEXT://linux_ip:9092
# retention period (7 days)
log.retention.hours=168
# allow automatic topic creation
delete.topic.enable=true

(4) Start Zookeeper and Kafka

# Start Zookeeper
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
# Start Kafka
nohup bin/kafka-server-start.sh config/server.properties &

(5) Verify that Kafka started successfully

# Check Zookeeper
ps -ef | grep zookeeper
# Check Kafka
ps -ef | grep kafka

(6) Test producing and consuming messages

# Change to Kafka directory
cd /usr/local/kafka
# Create a topic named test001
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.203.237:9092
# Start a producer
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.203.237:9092
# Start a consumer
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.203.237:9092

Result screenshots:

Additional Kafka query commands:

# List existing topics
bin/kafka-topics.sh --bootstrap-server 192.168.203.237:9092 --list
# Describe a specific topic
bin/kafka-topics.sh --bootstrap-server 192.168.203.237:9092 --topic test001 --describe
# List consumer groups
bin/kafka-consumer-groups.sh --bootstrap-server 192.168.203.237:9092 --list
# Delete a topic
bin/kafka-topics.sh --bootstrap-server 192.168.203.237:9092 --delete --topic test001

3. Common issues during setup

(1) Consumer cannot read data while producer works

# Locate Kafka log configuration
egrep -v "^*#|^$" server.properties

(2) Delete the kafka‑log folder and restart Kafka to clear topic data under /usr/local/kafka/kafka-logs