Master Kafka Deployment: Step-by-Step Configuration and Essential Commands

Kafka Deployment Configuration and Common Commands Summary

Deployment Configuration

1. Prepare deployment package (download yourself)

2. Configure Zookeeper

Edit conf/zoo.cfg with the following content:

dataDir=/data/vfan/zk/data/
dataLogDir=/data/vfan/zk/logs/
startLogDir=/data/vfan/zk/logs/
clientPort=2181
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=10.61.194.34:2801:3801
server.2=10.61.199.15:2802:3802
server.3=10.61.202.16:2803:3803
# server.A=B:C:D  where A is server id, B is IP, C is leader election port, D is quorum port
snapCount=20
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

Zookeeper cluster configuration as above; for a single node, comment out all server lines except server.1.

Start Zookeeper:

bin/zkServer.sh start
# ps to check process

2. Configure Kafka

Edit kafka/config/server.properties with the following settings:

broker.id=1
listeners=PLAINTEXT://10.153.204.28:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/vfan/kfk/logs/
num.partitions=3
default.replication.factor=3
offsets.topic.replication.factor=3
num.recovery.threads.per.data.dir=1
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.61.194.34:2181,10.61.199.15:2181,10.61.202.16:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

In cluster mode, modify broker.id and listeners in each broker’s config; set a single Zookeeper address for a single‑node setup.

Start Kafka:

bin/kafka-server-start.sh -daemon config/server.properties
# ss -tnlp|grep 9092 to check port

Common Commands Summary

Topic Operations

# List all topics
./kafka-topics.sh --zookeeper localhost:2181 --list

# Describe topics
./kafka-topics.sh --zookeeper localhost:2181 --describe

# Describe a specific topic
./kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

# Create a topic with 3 partitions and 3 replicas
./kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 3 --partitions 3

# Increase partitions
./kafka-topics.sh --alter --zookeeper localhost:2181 --topic test --partitions 3

# Delete a topic (enable delete.topic.enable=true)
./kafka-topics.sh --zookeeper localhost:2181 --delete --topic test

# Get message count per partition
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --time -1 --topic test

Produce and Consume

# Produce
./kafka-console-producer.sh --broker-list 10.153.204.28:9092 --topic test

# Consume from beginning
./kafka-console-consumer.sh --bootstrap-server 10.153.204.28:9092 --topic test --from-beginning

Ensure the broker address in producer/consumer matches the zookeeper.connect and listeners settings; otherwise you will see connection warnings.

Consumer Group Operations

# List consumer groups
./kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list

# Describe a consumer group
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group test-consumer

# Reset offsets
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupName --reset-offsets --to-offset 1000 --topic topicName --execute

Adjust Default Partition and Replication Settings

# In server.properties
num.partitions=3
default.replication.factor=3
offsets.topic.replication.factor=3

Change Topic Partition Count

Current topic “guoqing” has 1 partition and 1 replica.

# Increase partitions to 3
./kafka-topics.sh --alter --zookeeper localhost:2181 --topic guoqing --partitions 3
# Verify
./kafka-topics.sh --zookeeper localhost:2181 --describe --topic guoqing

Note: Partition count can only be increased, not decreased.

Increase Replication Factor

Create a JSON file describing the desired replica assignment:

vim guoqing.json
{
  "version": 1,
  "partitions": [
    {"topic":"guoqing","partition":0,"replicas":[1,2,3]},
    {"topic":"guoqing","partition":1,"replicas":[2,1,3]},
    {"topic":"guoqing","partition":2,"replicas":[3,2,1]}
  ]
}

Execute reassignment:

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/guoqing.json --execute

Verify progress:

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/guoqing.json --verify

Final topic description shows three partitions with replication factor three.