How to Build Single‑Node, Pseudo‑Cluster, and Distributed Kafka Environments on Ubuntu

1. Single‑Node Environment Setup

Install Zookeeper and Kafka on Ubuntu Server 16.04. Download the packages from the official sites, extract them, and start Zookeeper using the bundled configuration:

bin/zookeeper-server-start.sh config/zookeeper.properties

When the log shows the startup message, Zookeeper is running. Then start Kafka:

bin/kafka-server-start.sh config/server.properties

Create a test topic:

bin/kafka-topics.sh --create --zookeeper localhost:2181 \ --replication-factor 1 --partitions 1 --topic testTopic

Send a test message from the console producer and consume it with the console consumer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \ --topic testTopic --from-beginning

2. Pseudo‑Cluster (Multi‑Broker on One Machine) Setup

Copy the default server.properties file twice and edit each copy to assign a unique broker.id, listener port, and log directory:

config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2

Start both brokers in the background:

bin/kafka-server-start.sh config/server-1.properties &

bin/kafka-server-start.sh config/server-2.properties &

Create a replicated topic with a replication factor of 3 (even though only two brokers are running for demonstration):

bin/kafka-topics.sh --create --zookeeper localhost:2181 \ --replication-factor 3 --partitions 1 --topic my-replicated-topic

Produce and consume messages using the same console commands as in the single‑node setup.

3. Distributed Cluster Setup (Multiple Machines)

Assume three servers for Zookeeper and three for Kafka. Install Zookeeper on each server, copy zoo_sample.cfg to zoo.cfg, and adjust the data directory. Start each Zookeeper instance with:

./bin/zkServer.sh start

Verify the processes with jps. Install Kafka on each server, edit /conf/server.properties to set a unique broker.id, log directory, and the zookeeper.connect list (comma‑separated host:port entries). Then start each broker:

bin/kafka-server-start.sh config/server.properties

After all brokers are running, create topics, produce, and consume messages just as in the previous sections.

4. Code Test

The accompanying Maven project includes the required pom.xml, Log4j configuration, a simple producer class, and a consumer class. Build the project with Maven, run the producer to send messages, and observe the consumer receiving them.

Source code can be downloaded from the provided links or from the official Apache Kafka source distribution.