Master Kafka: Essential Commands for Starting, Managing Topics, and Messaging

1. Start Kafka Service

Run the following command to launch the Kafka broker in daemon mode using the server configuration file:

bin/kafka-server-start.sh -daemon config/server.properties

2. Stop Kafka Service

Stop the running Kafka broker with:

./kafka-server-stop.sh

3. Create a Topic

Use the kafka-topics.sh script to create a new topic. Example creates a topic named test0 with specific settings:

bin/kafka-topics.sh --create \
    --topic test0 \
    --zookeeper 127.0.0.1:2181 \
    --config max.message.bytes=12800000 \
    --config num.partitions=5 \
    --replication-factor 1

4. List All Topics

Display every topic known to the cluster:

bin/kafka-topics.sh --list --zookeeper localhost:9092

5. Describe Topic Details

Show detailed information for all topics, including partition count and replication factor:

bin/kafka-topics.sh --describe --zookeeper cdh-worker-1:2181/kafka

6. View Partition and Replica Information

Inspect the partitions and replica assignment for a specific topic:

bin/kafka-topics.sh --describe \
    --zookeeper 127.0.0.1:2181 \
    --topic test0

7. Delete a Topic

Remove an existing topic from the cluster:

bin/kafka-topics.sh --delete \
    --zookeeper 127.0.0.1:2181 \
    --topic test0

8. Send Messages (Producer)

Publish messages to a topic using the console producer:

./kafka-console-producer.sh \
    --broker-list localhost:9092 \
    --topic test

9. Receive Messages (Consumer)

Consume messages from the beginning of the topic:

./kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic test \
    --from-beginning

Consume only new messages arriving after the consumer starts:

./kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic test

Parameter Explanations

--topic : Name of the topic (e.g., test0).

--zookeeper : Zookeeper connection string, must match the zookeeper.connect setting in server.properties.

--config : Topic‑specific configuration options.

--partitions : Number of partitions for the topic.

--replication-factor : Number of replicas for each partition (default is 1).