Unlocking Kafka: Deep Dive into Architecture, Reliability, and Deployment

Kafka Architecture Overview

In Kafka, messages are categorized by topic ; producers write to topics and consumers read from them. A topic is a logical concept, while a partition is a physical log file storing producer data with offsets.

Each partition is split into multiple segment files, each consisting of an .index file and a .log file. Files are stored in directories named topicName-partitionId. Offsets are used for indexing and data retrieval.

Example: a topic test with three partitions creates directories test-0 , test-1 , test-2 .

Producers receive acknowledgments (acks) from the leader of each partition. Kafka offers three acks levels controlled by request.required.acks:

0 – No acknowledgment, highest throughput, lowest reliability.

1 – Leader acknowledgment only (default).

-1 / all – All in‑sync replicas must acknowledge, highest reliability.

Kafka ensures data consistency using LEO (Log End Offset) and HW (High Watermark). Follower failures are handled by removing them from the ISR set and resynchronizing from HW; leader failures trigger a new leader election and log truncation to HW.

Deployment Guide: Zookeeper, Kafka, Filebeat, ELK

Prepare the environment with nodes for Elasticsearch, Kibana, Logstash, Apache/Nginx/MySQL, Filebeat, and three Zookeeper/Kafka brokers.

node1: 192.168.67.11   elasticsearch  kibana
node2: 192.168.67.12   elasticsearch
apache: 192.168.67.10   logstash  apache/nginx/mysql
Filebeat node: 192.168.67.13   Filebeat
zk‑kfk01: 192.168.67.21   zookeeper, kafka
zk‑kfk02: 192.168.67.22   zookeeper, kafka
zk‑kfk03: 192.168.67.23   zookeeper, kafka

Start services, configure Filebeat to ship logs to Kafka, and create a Logstash pipeline that reads from the httpd topic and writes to Elasticsearch indices httpd_access-* and httpd_error-*.

output.kafka:
  enabled: true
  hosts: ["192.168.67.21:9092","192.168.67.22:9092","192.168.67.23:9092"]
  topic: "httpd"

Run Logstash with the pipeline configuration, adjust --path.data if path conflicts occur, and verify data in Kibana by creating an index pattern httpd_access-*.