How to Install and Test Kafka on CentOS: A Step‑by‑Step Guide

Kafka

Distributed publish‑subscribe type MQ High throughput Messages are unordered globally; ordering is guaranteed only within the same partition Version naming: kafka_2.11-0.10.1.1.tgz (2.11 is the Scala version, 0.10.1.1 is the Kafka version) Requires a JVM Concepts

Broker: a Kafka server; each server is a broker Topic: a category of messages Partition: one or more physical divisions of a topic Offset: unique identifier of a message within a partition Producer: message publisher Consumer: message subscriber Consumer Group: a concept unique to subscribers, defaulting to a single group

Install Kafka on CentOS

Apache project list

Install Zookeeper

Use an external Zookeeper instead of the one bundled with Kafka; download it with wget.

Use wget to download Use tar to extract Default port: 2181

wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

tar -zxvf zookeeper-3.4.9.tar.gz

Configure zoo.cfg :

cp zoo_sample.cfg zoo.cfg

vim zoo.cfg

Configure data directories:

dataDir=/data/www/data/zookeeper

dataLogDir=/data/www/logs/zookeeper

Start Zookeeper:

./zkServer.sh ../conf/zoo.cfg

Install Kafka

Use wget to download Use tar to extract Default port: 9092

Modify config/server.properties :

Port: 9092 External access: advertised.host.name=xxx.xx.xxx.xx (server's external IP) Log directory: log.dirs=/tmp/kafka-logs Partitions: num.partitions=1 Zookeeper connection: zookeeper.connect=localhost:2181 Zookeeper timeout: zookeeper.connection.timeout.ms=6000

Start Kafka:

./kafka-server-start.sh ../config/server.properties &

Test

Create a topic Start a producer Start a consumer

Create a topic:

./kafka-topic.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic

List topics:

./kafka-topics.sh --list --zookeeper localhost:2181

Start a producer:

./kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic

Start a consumer:

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopic --from-beginning

Send messages (producer) and view them (consumer) – screenshots illustrate the process:

Possible Errors

If Kafka and your application run on different servers, you may encounter connection errors; bind Kafka to the correct host:

host.name=xx.xx.xxxx.xx hostname should be the internal network address

Code

Refer to the GitHub repository for the full source code.