Visualizing Kafka: Core Concepts, Topics, Partitions, and Architecture Explained

Preface

Kafka is a mainstream streaming system with many concepts; this article uses diagrams to clarify its core ideas for a clear mental model.

Basics

Kafka is a stream‑processing system that enables backend services to communicate easily and is a common component in micro‑service architectures.

Producers and Consumers

Producer services send messages to Kafka, while Consumer services listen for and receive those messages. A single service can act as both producer and consumer.

Topics

A Topic is the destination address for producer messages and the listening target for consumers. A service can listen to or send messages on multiple topics.

Kafka introduces the concept of a consumer group, which is a set of services acting together as a single consumer. When a consumer group receives a message, Kafka routes the message to only one service within the group, enabling load balancing and easy scaling.

Partitions

A Topic consists of multiple queues called partitions, which allow the topic to scale. When a producer sends a message, it is routed to a specific partition; consumers listen to all partitions. By default, Kafka uses a round‑robin strategy to assign messages to partitions, but you can configure a topic so that related messages go to the same partition, preserving order.

Architecture

Kafka runs as a cluster, with ZooKeeper as a critical component that manages all topics and partitions across physical nodes.

Topics and partitions are stored on node machines; ZooKeeper maintains their metadata. Each partition has a leader that receives producer messages and replicates them to follower replicas, ensuring reliability and fault tolerance.