Skip to content
Tagged articles

log segments

3 articles · Page 1 of 1
Architecture Pony
Architecture Pony
Nov 15, 2026 · Backend Development

Kafka Storage Deep Dive: Append-Only Logs, Sparse Indexes & Compact Traps

This article dissects Kafka's on-disk storage architecture, explaining partition segments, sparse indexes, page cache reliance, flush policies, delete vs. compact cleanup, and real-world incidents like index rebuild storms and misconfigured retention, with production configs and capacity formulas.

Kafkacapacity planningconfiguration
0 likes · 50 min read
Kafka Storage Deep Dive: Append-Only Logs, Sparse Indexes & Compact Traps
Architect
Architect
Aug 3, 2024 · Big Data

Kafka Storage Architecture Design: Deep Analysis and Implementation

This article thoroughly examines Kafka's storage architecture, analyzing its design motivations, storage mechanisms, log formats, partitioning, indexing, cleanup strategies, and performance optimizations such as sequential log writes, sparse indexing, page cache, and zero‑copy, providing insights applicable to other storage systems.

Kafkalog segmentssparse index
0 likes · 22 min read
Kafka Storage Architecture Design: Deep Analysis and Implementation
ITPUB
ITPUB
Oct 26, 2022 · Big Data

Why Kafka Stores Data the Way It Does: Inside Its Architecture

This article provides an in‑depth technical analysis of Kafka’s storage architecture, covering its design goals, storage mechanisms, log segment layout, sparse indexing, log cleanup policies, and the performance techniques such as sequential writes, page cache, and zero‑copy that enable high‑throughput streaming.

Log Cleanupbig datalog segments
0 likes · 22 min read
Why Kafka Stores Data the Way It Does: Inside Its Architecture