Kafka Core Concepts: Basics, Producers/Consumers, Topics, Partitions, and Architecture

Kafka is a mainstream streaming platform that enables backend services to communicate easily and is a common component in micro‑service architectures.

Basics

Kafka functions as a flow‑processing system, and this section introduces its core concepts to give readers a clear mental model.

Producer‑Consumer

Producer services send messages to Kafka, while Consumer services listen to Kafka to receive messages. A single service can act as both producer and consumer.

A service can simultaneously be a producer and a consumer.

Topics

Topics are the destination addresses for producers and the listening targets for consumers. A service can publish to and subscribe from multiple topics.

Kafka introduces the concept of a consumer‑group, which is a set of services acting as a single consumer. Messages are routed to one member of the group, enabling load balancing and easy scaling.

Messages remain in the topic queue after being consumed, allowing continuous processing and configurable retention periods.

Partitions

A topic is composed of multiple queues called partitions, which provide parallelism and ordering guarantees within each partition.

Producers send messages to a specific partition, typically using a round‑robin strategy, but can be configured to route related messages (e.g., from the same user) to the same partition to preserve order.

Consumers listen to all partitions of a topic.

Architecture

Kafka runs as a clustered system with ZooKeeper managing topics and partitions. Each partition is replicated across multiple broker nodes, with one replica acting as the leader and the others as followers.

The leader receives producer messages and replicates them to followers, ensuring durability and fault tolerance; if a node fails, other replicas continue serving data.

This design provides high reliability, scalability, and ordered processing within partitions.

For further reading, the original visual guide is available at https://timothystepro.medium.com/visualizing-kafka-20bc384803e7 .