How Does Kafka’s Cluster Architecture Enable Scalable Messaging?

Main content: 1. Kafka overall structure 2. Message production 3. Message consumption

Overall Structure

In Kafka, a topic is created, and a producer writes messages to the topic while a consumer reads messages from it.

Kafka operates as a clustered system; each topic is divided into multiple partitions, and each partition is evenly replicated across different servers, with the replication factor configurable.

Each partition has two roles: a leader and a follower.

The leader handles all read and write requests.

The follower provides fault tolerance and takes over as leader if the current leader fails.

Message Production

A producer writes data to a topic, which actually means writing to a specific partition; the producer decides which partition to use, often by round‑robin.

Messages are appended to a partition in order, each assigned a sequential offset.

Message Consumption

A consumer belongs to a consumer group, which may contain one or more consumers.

The group subscribes to a topic collectively; individual consumers do not subscribe directly.

When new messages appear in the topic, they are delivered to one consumer in the group, enabling load balancing when multiple consumers exist.

Consumers in the group can be different processes on the same machine or on different servers.

Consumers maintain their own read position; Kafka does not track offsets, so each consumer decides from which offset to start reading.

After messages are read, they are not deleted, allowing repeated reads; Kafka cleans up expired messages based on a configured retention time.

Summary

Kafka contains multiple topics, each split into several partitions that are evenly replicated across the cluster with designated leaders and followers; leaders handle read/write operations while followers provide failover.

Producers append messages sequentially to a chosen partition.

Consumers operate as part of a consumer group, sharing the load across partitions, deciding their own read offsets, and enabling scalable, fault‑tolerant message consumption.