Master Kafka Basics: Architecture, Core Concepts, and Hands‑On Python Experiments

What is Kafka

Kafka is a distributed streaming system that can publish and subscribe to messages like a message queue, offering fault tolerance and concurrent processing.

Basic Concepts

Kafka runs on a cluster of one or more servers. Messages are stored in topics, each message containing a key, value, and timestamp.

Producer – client that sends messages to a Kafka broker.

Consumer – client that reads messages from Kafka.

Topic – user‑defined category that links producers and consumers.

Partition – ordered queue within a topic; each message gets a unique offset.

Broker – a Kafka server; a cluster consists of multiple brokers.

Consumer Group – a set of consumers that share the consumption of a topic’s partitions.

Offset – the position of a message within a partition.

Distributed Architecture

Messages in a topic are spread across partitions. If a key is present, the key determines the partition; otherwise a round‑robin algorithm is used. Each partition is ordered, and only one consumer in a consumer group may read from a given partition at a time. Partitions are replicated across brokers; each partition has a leader broker and one or more follower brokers. All reads and writes go through the leader, which replicates data to followers.

Experiment 1: kafka‑python Producer and Consumer

Using the kafka‑python client, a simple producer sends messages to a test topic and a consumer reads them. The experiment shows the producer and consumer code, topic creation, and successful consumption of messages.

Experiment 2: Consumer Group Fault Tolerance

This experiment creates a topic with two partitions and two consumers belonging to the same consumer group. Each consumer reads from a different partition; when one consumer stops, the remaining consumer takes over the other partition, demonstrating fault tolerance.

Experiment 3: Offset Management

Kafka allows a consumer to commit its current offset. This experiment shows a consumer restarting and resuming consumption from the last committed offset, confirming reliable offset handling.

Conclusion

The article introduced Kafka’s core concepts and illustrated them with hands‑on experiments covering producer/consumer basics, consumer‑group fault tolerance, and offset management.