Master Kafka Basics: Topics, Partitions, Producers & Consumers Explained

Fundamentals

Kafka is a mainstream streaming platform with many concepts; the following diagrams help clarify its core ideas.

Producers and Consumers

A Producer service sends messages to Kafka, while a Consumer service listens for and receives those messages.

A single service can act as both producer and consumer.

Topics

A Topic is the destination address for messages sent by producers and the listening target for consumers.

Kafka also defines a consumer‑group, a set of services that together act as a single consumer. When a consumer group receives a message, Kafka routes the message to one service in the group, enabling load balancing and easy scaling.

Messages are stored in the Topic’s queue and remain immutable; they persist in the queue until configured otherwise.

Partitions

A Topic consists of multiple queues called Partitions, which facilitate Topic expansion.

When a producer sends a message, it is routed to a specific Partition of the Topic.

Consumers listen to all Partitions.

By default, the producer uses a round‑robin strategy to assign messages to Partitions, but Topics can be configured so that related messages share the same Partition, preserving ordering within that Partition.

Only messages within the same Partition are guaranteed to be ordered.

Architecture

Kafka runs as a clustered system; ZooKeeper is a critical component that manages all Topics and Partitions.

Each Partition has a Leader node that receives producer messages and replicates them to Followers, ensuring full data redundancy and fault tolerance.

Even if a node fails, the replicated data in other nodes prevents message loss.

Thank you for reading; hope this helps you understand Kafka.