Master Kafka: Install, Run a Single Node, Build a Cluster, and Use Kafka Connect

Main content: 1. Kafka installation and startup 2. Message production and consumption 3. Configuring and launching a cluster 4. Fault‑tolerance testing in the cluster 5. Importing data from a file and exporting it to another file

Single‑Node Example

Installation

tar -xzf kafka_2.10-0.10.1.1.tgz
cd kafka_2.10-0.10.1.1

Startup

> bin/zookeeper-server-start.sh config/zookeeper.properties
> bin/kafka-server-start.sh config/server.properties

Create a Topic

Open a new terminal window.

bin/kafka-topics.sh --create \
--zookeeper localhost:2181 \
--replication-factor 1 \
--partitions 1 \
--topic test

Send Messages

Open a new terminal window.

bin/kafka-console-producer.sh \
--broker-list localhost:9092 \
--topic test

Enter messages, for example:

hello world
hi

Consume Messages

Open a new terminal window.

bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test \
--from-beginning

The two messages you sent will be displayed:

hello world
hi

You can continue typing new messages in the producer terminal and they will appear instantly in the consumer.

Cluster Configuration

Create Two Server Config Files

> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties

Modify config/server-1.properties:

broker.id=1
listeners=PLAINTEXT://:9093
log.dir=logs/kafka-logs-1

Modify config/server-2.properties:

broker.id=2
listeners=PLAINTEXT://:9094
log.dir=logs/kafka-logs-2

Start the Cluster

> bin/kafka-server-start.sh config/server-1.properties &
> bin/kafka-server-start.sh config/server-2.properties &

Create a Replicated Topic (3 replicas)

bin/kafka-topics.sh --create \
--zookeeper localhost:2181 \
--replication-factor 3 \
--partitions 1 \
--topic my-replicated-topic

Send Messages to the Replicated Topic

bin/kafka-console-producer.sh \
--broker-list localhost:9092 \
--topic my-replicated-topic

Enter messages:

my test message 1
my test message 2

Consume Messages

bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--from-beginning \
--topic my-replicated-topic

Messages are retrieved successfully.

Fault‑Tolerance Test

# Get the process ID of server‑1
ps aux | grep server-1.properties

# Kill the process
kill -9 43116

After killing server‑1, read the messages again:

bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--from-beginning \
--topic my-replicated-topic

The same messages are still available, confirming fault tolerance.

Kafka Connect

Kafka Connectors enable integration with external systems such as file systems and databases.

The following experiment imports data from a file and exports it to another file.

Create a Test File for Import

echo -e "foo
bar" > test.txt

Start Connect in Standalone Mode

bin/connect-standalone.sh \
config/connect-standalone.properties \
config/connect-file-source.properties \
config/connect-file-sink.properties

The command prints logs until the connectors are ready.

View Exported Results

cat test.sink.txt

Output:

foo
bar

The data from test.txt has been successfully exported.

Process Analysis

The import and export behavior is defined by two configuration files.

config/connect-file-source.properties (source connector)

name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=test.txt
topic=connect-test

The file property points to test.txt, and the topic property sends the data to the connect-test topic.

config/connect-file-sink.properties (sink connector)

name=local-file-sink
connector.class=FileStreamSink
tasks.max=1
file=test.sink.txt
topics=connect-test

Here, file specifies the destination test.sink.txt, and topics tells the connector to read from connect-test.

To verify the topic content:

bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic connect-test \
--from-beginning

Result:

{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}

Appending a new line to test.txt: echo "Another line" >> test.txt Checking test.sink.txt now shows the new data:

foo
bar
Another line