Kafka: Usage Scenarios, Core Concepts, Installation, Configuration, and Practical Operations

Kafka is a distributed, partitioned, replicated messaging system originally developed by LinkedIn and now an Apache top‑level project; it excels at real‑time processing of large data streams for log collection, messaging, user activity tracking, and operational metrics.

Core concepts include topics, partitions, brokers, producers, consumers, and Zookeeper coordination. A producer sends messages to a topic, which are stored in ordered partitions identified by offsets; consumers read messages either individually (queue mode) or via publish‑subscribe using consumer groups.

Installation preparation requires JDK and Zookeeper. Example commands:

yum install java-1.8.0-openjdk* -y

wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.5.8/apache-zookeeper-3.5.8-bin.tar.gz
tar -zxvf apache-zookeeper-3.5.8-bin.tar.gz
cd apache-zookeeper-3.5.8-bin
cp conf/zoo_sample.cfg conf/zoo.cfg

Start Zookeeper:

bin/zkServer.sh start
bin/zkCli.sh

Kafka installation (2.4.1) :

wget https://mirror.bit.edu.cn/apache/kafka/2.4.1/kafka_2.11-2.4.1.tgz
tar -xvf kafka_2.11-2.4.1.tgz
cd kafka_2.11-2.4.1

Configure config/server.properties (broker.id, listeners, log.dir, zookeeper.connect) and start the broker:

bin/kafka-server-start.sh -daemon config/server.properties

Topic management examples:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic test_topic --partitions 3 --replication-factor 1

bin/kafka-topics.sh --list --zookeeper 192.168.65.60:2181

bin/kafka-topics.sh --delete --topic test --zookeeper 192.168.65.60:2181

Send and consume messages:

bin/kafka-console-producer.sh --broker-list 192.168.65.60:9092 --topic test_topic
> this is a msg

bin/kafka-console-consumer.sh --bootstrap-server 192.168.65.60:9092 --topic test_topic --from-beginning

Cluster expansion can be done by copying server.properties to create additional brokers (broker.id 1, 2) with distinct listeners and log directories, then starting them with the same Zookeeper address.

Java client example (Maven dependency org.apache.kafka:kafka-clients:2.4.1 ) shows producer configuration, message serialization with FastJSON, and asynchronous callback handling.

The guide also explains consumer groups, single‑ vs multi‑cast consumption, partition rebalancing, leader election, and how offsets are maintained per consumer.