Understanding Kafka Topic Architecture: Partitions, Replication, and Failover

1. Kafka Topic, Log, Partition

Kafka Topic (topic) is a named stream of records, and Kafka stores each record in a log file.

A topic is divided into multiple Partitions .

Kafka distributes the log's partitions across multiple servers.

Splitting a topic into partitions improves speed, scalability, and storage capacity.

A topic follows a publish/subscribe model; a topic can have zero or many subscribers (consumer groups).

Kafka Topic Partitions

A topic contains multiple partitions, and each partition holds many records.

How is a specific record stored in a partition?

If a record has a key, the key determines the partition.

If a record has no key, Kafka uses round‑robin to assign a partition.

Multiple partitions give a topic scalability across servers, improving producer write performance and allowing parallel consumer consumption; the parallelism limit equals the number of partitions.

Records within a partition are ordered. When partitioning by key, records with the same key reside in the same partition, which is useful for log replay scenarios.

Kafka replicates each partition to ensure high availability and enable failover.

Kafka Record Ordering

Kafka guarantees ordering only within a partition, not across the entire topic.

A partition is an ordered, immutable sequence of records.

Kafka treats a partition as a structured commit log, continuously appending records.

Each record in a partition receives a sequential number called an "offset," indicating its position.

Topic partitions allow Kafka logs to scale beyond a single server's capacity.

Partitions must fit on their host server, but a topic can have many partitions, allowing it to span multiple servers.

A partition is a parallel unit; only one consumer in a consumer group can read from a partition at a time.

If a consumer in a group stops, Kafka reassigns its partitions to other consumers in the group.

Kafka Topic Partition Replication

Kafka can replicate partitions across servers; the replication factor is configurable and provides fault tolerance.

Each partition has multiple copies: one leader and one or more followers. The leader handles all read and write requests for the partition.

Followers replicate records from the leader and monitor the leader's health.

2. Replication: Leader, Follower, and ISR

Kafka uses ZooKeeper to elect a leader for each partition.

The server hosting the partition leader processes all read/write requests; write requests are replicated from the leader to followers.

Among followers, those that stay in sync with the leader are called ISR (in‑sync replicas). If the leader fails, a new leader is elected from the ISR.

A record is considered "committed" only after it has been replicated to all ISR members; only committed records are visible to consumers.

3. Frequently Asked Questions

What is ISR?

An ISR is a follower that is synchronously replicating the leader.

If the leader fails, only ISR members are eligible to become the new leader.

Consumer‑Partition Relationship

A consumer can be assigned multiple partitions, but a partition can be consumed by only one consumer within a consumer group at a time.

If a topic has a single partition, only one consumer can read from it.

What are Leader and Follower?

The leader handles all read/write operations for its partition; followers replicate the leader.

How does Kafka handle consumer failover?

If a consumer in a group fails, its assigned partitions are redistributed to other consumers in the group.

How does Kafka handle broker failover?

When a broker fails, Kafka reassigns the leadership of its partitions to other brokers.