Kafka Storage Mechanism and Reliability Guarantees

Kafka Storage Mechanism

Kafka organizes data by topics, each divided into partitions that may have multiple replicas. Within each partition, data is stored in a series of equal‑sized .log and .index segment files. The segment naming starts at 0 and each subsequent segment is named after the offset of the last message in the previous segment.

Segment Structure

A .log file holds the raw message bytes, while the accompanying .index file stores offset‑to‑position mappings, enabling efficient lookup of a message's location within the log.

Reading Data

To read a specific offset, the client determines which segment contains the offset, consults the segment's index to find the start position in the .log file, and then reads the message according to the fixed record format.

Reliability Guarantees

Replica Sets (AR, ISR, OSR)

All replicas of a partition form the Assigned Replicas (AR) list, which is split into In‑Sync Replicas (ISR) and Out‑of‑Sync Replicas (OSR). Only ISR members have fully replicated the leader's data.

High Watermark (HW) and Log End Offset (LEO)

LEO marks the latest offset written to the leader. HW marks the highest offset that has been replicated to all ISR members; only data before HW is visible to consumers.

Producer Acknowledgment Levels

Kafka provides three acknowledgment settings via request.required.acks:

0 – fire‑and‑forget, highest throughput, lowest durability.

1 – leader acknowledges after writing to its log.

-1 (all) – leader waits for all ISR replicas to replicate before acknowledging, offering the strongest durability.

Additional safety can be enforced with min.insync.replicas to require a minimum number of ISR members.

Leader Election

If the current leader fails, a new leader is chosen from the ISR. Configuration unclean.leader.election.enable controls whether a non‑ISR replica may be elected (higher availability, lower safety) or not (higher safety, lower availability).

Overall Delivery Guarantees

Kafka can guarantee at‑least‑once delivery by default; achieving exactly‑once semantics requires additional idempotent producer and transactional configurations.