Master Kafka Basics: Visual Guide to Topics, Partitions, and Architecture

Kafka is a mainstream streaming system with many concepts; the following visual guide organizes its core concepts for clear understanding.

01 Basics

Kafka is a stream processing system (also called a message queue) that enables backend services to communicate easily and is a common component in microservice architectures.

02 Producer and Consumer

Producer services send messages to Kafka, while consumer services listen to Kafka to receive messages.

A single service can act as both producer and consumer.

03 Topics

A Topic is the destination address for producers and the listening target for consumers.

One service can listen to and send messages to multiple Topics.

Kafka includes the concept of a consumer group, which is a set of services acting as a single consumer.

When a consumer group receives messages, Kafka routes each message to one service within the group, enabling load balancing and scalable consumption.

Messages are stored in the Topic queue and are immutable; they remain in the queue even after being consumed, depending on Kafka’s configuration.

04 Partitions

A Topic consists of multiple queues called Partitions, which facilitate Topic scalability.

When a producer sends a message, it is routed to a specific Partition of the Topic, typically using a round‑robin strategy, though custom routing can ensure that related messages share the same Partition to preserve ordering.

Consumers listen to all Partitions and consume events from each.

Ensuring that related messages stay in the same Partition provides ordering guarantees, as only messages within a single Partition are ordered.

05 Architecture

Kafka runs as a clustered system; ZooKeeper is a critical component that manages Topics and Partitions across physical nodes.

Each Partition has a Leader that receives messages from producers; Followers replicate the Leader’s data, ensuring full message availability and fault tolerance.

Even if a node fails, messages are not lost.

For further reading on Kafka and message queues, refer to additional resources.