Kafka Storage Deep Dive: Append-Only Logs, Sparse Indexes & Compact Traps
This article dissects Kafka's on-disk storage architecture, explaining partition segments, sparse indexes, page cache reliance, flush policies, delete vs. compact cleanup, and real-world incidents like index rebuild storms and misconfigured retention, with production configs and capacity formulas.
Introduction
The previous articles covered cluster, producer, and consumer — all about data in motion. This piece drills down to where data lands on disk. Kafka's storage design is the foundation of its entire story: why a broker can host hundreds of thousands of partitions without stalling, why reading one message from a billion returns in seconds, and why restarts don't lose data yet don't wait for fsync. The scariest production faults — disk full, index corruption, messages "disappearing" after compaction — all originate here because problems in this layer don't error; they just silently slow down until they explode collectively some early morning.
Three Main Threads
Each partition is a directory plus a series of append-only segment files — no B-trees, no page structures, everything serves sequential I/O.
Fast reads rely on sparse indexes, but indexes are bound to data; when segments roll, delete, or recover, indexes must follow — the pitfalls are all in "following along."
Log cleanup has two rule sets: delete and compact. Misconfigured levels or keys cause sudden disk explosions or "missing messages" weeks later.
Incident Case Studies
Incident 1: Restart Once, Broker Stuck for 40 Minutes
During a rolling restart, one broker failed to register. Logs showed "Recovering from failure... loading log" with sustained 200 MB/s disk reads. Root cause: an OOM kill the previous week left a 90 GB segment file inconsistent with its .index. Kafka detected this and discarded the old index, rebuilding it by scanning the entire segment. One 90 GB partition is fine, but this broker hosted 200 partitions totaling 2 TB; scanning took 40 minutes. During that time all leader partitions on the node were unavailable, triggering a cascade of leader elections and client reconnections.
Incident 2: Disk at 92%, but 7-Day Retention Didn't Fire
Alert showed 92% disk usage. Ops assumed retention.ms=7 days would clean up. Investigation revealed retention.ms was set at the cluster broker level, but the affected topics were created via Confluent client with explicit retention.ms=-1 (retain forever). Topic-level config overrides cluster-level — log files kept growing weekly, deletion threads ignored them. Eventually 20 partition leaders were forced to step down, slowing producer writes.
Incident 3: After Compaction, Business Says "Messages Lost"
A compacted topic stored config snapshots; upstream did a full dump every 10 minutes. Three months later a config item couldn't be found. The key used a timestamp suffix, so each dump had a different key. Compaction keeps the latest value per key — old keys are never cleaned, but new keys don't overwrite old ones. The result wasn't "latest value per config" but "latest value per historical key," data accumulated, and queries returned three-month-old values. Compaction wasn't wrong; the mistake was treating it as an "auto-deduplicating table."
Technical Analysis: A Partition's Full On-Disk Lifecycle
1.1 One Partition = One Directory + a Series of Segment Files
A topic-partition on disk is a directory named like order-event-0. No database-style "table structure" exists — only segments. Each segment is a trio: xxx.log (message payload), xxx.index (offset index), xxx.timeindex (timestamp index). Filename is the global offset of the first message in that segment, e.g., 00000000000000381230.log means this segment starts at offset 381230. Writes always append to the last active segment . When the active segment reaches log.segment.bytes (default 1 GB) or ages past log.segment.ms, it "rolls" and a new segment opens.
Three often-ignored facts: (1) segment.bytes and segment.ms — whichever hits first triggers a roll; the time parameter only matters for low-throughput topics. A topic at 10 msg/s may not roll for a year on the 1 GB threshold, packing all history into one segment, making retention deletion impossible (deletion granularity is whole segments). (2) Deletion granularity is segment, not message: "7-day expiry" works at segment boundaries — only when the earliest segment's latest timestamp expires can the whole segment be deleted. (3) The partition number in the directory name is the local replica ID on that broker; the same partition on different brokers has different directories — don't treat them as three copies of the same data for reconciliation.
1.2 Sequential Writes: Why Append-Only Beats B-Trees
Databases use B-trees to support "update any row"; Kafka only needs append, so it chose the simplest structure: files. HDD sequential write ~100 MB/s, random write only a few MB/s; SSD sequential write sustains 2 GB/s+ while random write drops 10x due to write amplification. Append-only brings two hidden bonuses: zero lock contention on write path (only active segment is written, old segments are read-only, naturally fitting multi-reader single-writer cache semantics) and OS handles merging (page cache batches small writes into large flushes).
The trade-off is explicit: no updates . To "modify a message," Kafka appends a new message with the same key and relies on compaction to purge the old one (Section 1.7). This "append-only + deferred cleanup" combo is the fundamental divide between Kafka and traditional MQ storage — it turns a message queue into a replayable log , shifting the storage problem from "how to update" to "how to delete."
1.3 Sparse Index: Four Reads to Locate One in a Billion
Indexing every message would consume disk. Kafka records an index entry only every index.interval.bytes (default 4 KB) of data — that's "sparse." To find offset 500, binary search in .index finds the largest index entry ≤ 500 (say it says offset 480 at physical offset 2 MB), then sequentially scan the log file from 2 MB, comparing offsets, at most 4 KB until hit. The whole lookup: one in-index binary search (in memory-mapped pages) + one seek read + a tiny sequential read.
.timeindexworks similarly but maps timestamps to offsets. It underpins time-based replay ( offsetsForTimes) and retention's "segment latest timestamp" check. A subtle detail: timeindex records "first message after this timestamp," while segment deletion uses the segment's maximum timestamp — different semantics that often confuse "why wasn't this deleted" investigations.
1.4 Page Cache: Why Kafka Doesn't Build Its Own Cache
Traditional MQs keep message caches in JVM heap — cost: hotter cache = worse GC, restart loses cache, two copies (cache + disk) require manual consistency. Kafka pushes everything to the OS: writes go to page cache first, reads hit page cache first . Hot data produced and immediately consumed never touches disk — the other half of Kafka's low latency (first half is producer batching).
Two direct production takeaways: (1) JVM heap for Kafka needn't be large — 6 GB is enough; memory should be left for filesystem page cache. On a 64 GB machine, give Kafka 6 GB heap, the remaining 50+ GB becomes page cache potential. (2) Tail reads fast, historical reads slow : consumers reading at the latest offset hit cache entirely; but a replay from three days ago (e.g., upstream data fix) evicts historical data into page cache, causing IO spikes and potentially evicting other partitions' hot data — one large-scale replay can drag down the whole broker .
1.5 Flushing: Reliability Comes from Replicas, Not fsync
Many think acks=all prevents loss via flushing — wrong. Kafka ignores fsync by default ( log.flush.interval.messages and log.flush.interval.ms disabled). Data returns success while in page cache. Power loss loses page cache data? No problem: the unflushed part on this broker exists on other replicas — reliability is guaranteed by multi-replica + ISR (detailed in Article 05). Flushing only affects "how much log to replay on single-node restart." That's why don't enable log.flush.interval.messages=1 : it turns every message into an fsync, throughput drops an order of magnitude, and the "single-node power-loss safety" is meaningless in a multi-replica architecture.
The real risk window: acks=1 and this leader crashes before flushing — data lost in page cache with no replica backup. So " acks=1 + single replica" is the truly naked config; under multi-replica + acks=all, keeping flush defaults is correct.
1.6 Delete Cleanup: Three Dimensions and Three Pitfalls
Delete mode judges on three dimensions : log.retention.ms/minutes/hours (time, default 7 days), log.retention.bytes (per-partition size, default unlimited), and log.segment.bytes indirectly sets deletion granularity. Cleaner thread scans every log.retention.check.interval.ms (default 5 min), deleting whole segments whose earliest segment's latest timestamp has expired. Pitfall 1: timestamp source — if upstream sets message timestamp to business time (e.g., report date), backfilling historical data carries old timestamps; cleaner immediately deems segments expired and deletes them — backfilled data deleted before consumption . Pitfall 2 is Incident 2's config hierarchy : topic-level overrides broker-level; dynamic broker changes don't affect existing topics. Pitfall 3: retention.bytes is per-partition , not topic total — don't multiply wrong when capacity planning.
1.7 Compaction: Tombstones and "Latest Value Per Key"
Compact mode ( cleanup.policy=compact) doesn't delete by time but by key : only the latest message per key is kept. It's the storage foundation for "changelog + snapshot" semantics: upstream sends full or incremental updates, downstream can always read a converged current state from any position. Two key mechanisms: (1) Tombstones : a message with null value means "delete this key"; compaction retains it for delete.retention.ms (default 24 hours) so downstream consumers see the deletion event, then purges it. (2) Compaction trigger: log.cleaner.min.cleanable.ratio (default 0.5) — a segment's "dirty" portion must exceed half of cleanable space to be worth cleaning. Thus low-throughput compact topics don't clean in real-time; an observation window exists where "new value sent but old segment still returns old value."
Three hard constraints of compaction, violations cause incidents: (1) Key cannot be null — compact topics reject keyless messages (actually good, forces upstream discipline). (2) Order determined by key — after compaction, physical order no longer matches write order; all historical ordering for a key is lost. Scenarios needing order (e.g., audit) cannot use compact. (3) Cleaning only targets sealed segments — active segment never cleaned; a compact topic's "latest value" is only complete when consumption reaches near the active segment.
Production Practice: Two Complete Configs and a Recovery Postmortem
2.1 Config 1: Broker-Side Full Storage Config
This extends the server.properties storage section from Article 01, each line annotated with the pitfall it prevents. Dual-disk complete version, production-ready.
# Storage dirs: comma-separated for multi-disk, Kafka spreads partitions automatically
log.dirs=/data1/kafka,/data2/kafka
# Disable auto-recreate dirs: missing dir requires manual intervention
log.dirs.failure.recovery=false
# Segment rolling: 512MB finer granularity, smoother delete/migration
log.segment.bytes=536870912
# Low-throughput safety: must roll at least every 24h
log.segment.ms=86400000
# Index: per-segment index cap 16MB, enough for 512MB segment
log.index.size.max.bytes=16777216
# Sparsity density: default 4KB, hot-read scenarios can go to 8KB
log.index.interval.bytes=4096
# Retention: cluster-level defaults; existing topics use topic-level (see 2.2 check script)
log.retention.hours=72
log.retention.check.interval.ms=300000
# No per-partition size limit, rely on time control
#log.retention.bytes=-1
# Flushing: keep defaults off, rely on replicas for durability
#log.flush.interval.messages=Long.MAX
#log.flush.interval.ms=null
# Compression: server no longer compresses, respect client
compression.type=producer
# Cleaner threads: 0.5GB memory + 2 threads
log.cleaner.enable=true
log.cleaner.dedupe.buffer.size=536870912
log.cleaner.threads=2
# Clean when dirty ratio exceeds half; lower for low-throughput topics
log.cleaner.min.cleanable.ratio=0.25
# Tombstone retention: give downstream 1 day to see delete event
log.cleaner.delete.retention.ms=864000002.2 Config 2: Compact Topic Config & Health-Check Script
Compact config scattered across topic and broker levels, easiest to miss. Below: full command to create a config-snapshot topic, plus a periodic health-check script — the answer to Incident 2's "did retention actually take effect?"
# Create compact topic: must include min.insync
kafka-topics.sh --bootstrap-server k0:9092 \
--create --topic config-snapshot \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.insync.replicas=2 \
--config segment.ms=3600000 \
--config delete.retention.ms=86400000 \
--config min.cleanable.dirty.ratio=0.2
# Check cleaner progress: cleaner metrics
kafka-run-class.sh kafka.tools.\
GetOffsetShell --topic config-snapshot
kafka-configs.sh --bootstrap-server k0:9092 \
--describe --entity-type topics \
--entity-name config-snapshot#!/bin/bash
# storage_check.sh: storage health-check, cron every 10 min
BOOT=k0:9092; DIRS="/data1/kafka /data2/kafka"
ALERT_DISK=85 # disk usage threshold %
ALERT_SEG=200 # per-partition segment count threshold
# 1. Disk usage (also watch inode)
for d in $DIRS; do
u=$(df -P $d | awk 'NR==2{print $5}' | tr -d '%')
i=$(df -iP $d | awk 'NR==2{print $5}' | tr -d '%')
[ $u -ge $ALERT_DISK ] && echo "[D] $d $u%"
[ $i -ge $ALERT_DISK ] && echo "[I] $i%"
done
# 2. Topic-level retention override check
# Prevent Incident 2: who quietly set retention to forever
kafka-configs.sh --bootstrap-server $BOOT \
--describe --entity-type topics --all \
| grep -B2 'retention.ms=-1'
# 3. Compact topic keyless message check (upstream discipline)
# 4. Segment count: too many = rolling too frequent, too few = can't delete
# 5. Cleaner backlog: log-cleaner-recents-per-sec and dirty-bytes below alert line
echo "check #3-5 see JMX kafka.log:type=LogCleanerManager"2.3 Capacity Math: Retention × Throughput × Replicas
Capacity formula is simple: Disk = per-partition write rate × retention seconds × replica count ÷ disk count , times 1.3 headroom (compact topics add extra for min.cleanable.ratio "uncleaned dirty space"). The real number to calculate: worst-case rebound space . Downstream consumer groups stop for a day — lag doesn't consume disk; but upstream keeps producing, disk grows linearly at write rate. A 72-hour retention cluster, full stop for a day then catch-up, hits the boundary exactly. So capacity must plan for retention + one day's write volume , not just retention. Water-level alerts at 70% and 85%: 70% triggers manual assessment, 85% triggers retention reduction or expansion playbook.
2.4 Postmortem: Compressing 40-Minute Index Rebuild to 4 Minutes
Incident 1's core contradiction: recovery speed . Index rebuild time ≈ partition data total ÷ sequential disk read speed. 2 TB ÷ 200 MB/s ≈ 3 hours (that day 40 min because many segments had page cache). Three improvements by priority: (1) Shorten segment time : segment.bytes from 1 GB to 512 MB — rebuild works per segment, smaller blast radius; also makes deletion granularity finer. (2) Manual recovery ordering : before restart, migrate critical topic partition leaders away ( kafka-reassign) so business traffic fails over first, broker scans slowly. (3) Eliminate inconsistency source : OOM kill more likely causes half-written logs (message written but index not updated) than kill -9. Give JVM -XX:+ExitOnOutOfMemoryError so process suicides on OOM and runs normal shutdown — safer than "struggling to stay alive." Post-incident, "single broker load > 10 min" became a monitored metric; rebuild exceeding threshold alerts immediately, not waiting for business to notice leader unavailability.
2.5 OS Layer: Filesystem & Kernel Params
Half of storage performance ceiling lives in OS. Pick XFS — outperforms ext4 on large-file appends and concurrent directory ops (gap obvious with hundreds of thousands of partition directories). Mount with noatime — Kafka reads logs don't need access time; dropping it saves a metadata write per read. Three kernel params: vm.dirty_background_ratio / vm.dirty_ratio control page cache dirty-page flush aggressiveness; defaults cause big flush stalls at write peaks — usually lower (background 5%, total 15%); vm.swappiness=1 — page cache swapped out = disaster; vm.max_map_count must be raised (Kafka heavily mmaps index files, one mapping per index, default 65535 fills up in large clusters). Cloud disk users add: don't buy highest IOPS tier — Kafka is sequential I/O; random IOPS money mostly wasted; look at throughput tier only.
Core Source Code: Four Key Locations
3.1 OffsetIndex.lookup: Binary Search Index
// OffsetIndex.scala (simplified)
def lookup(target: Long): OffsetPosition = {
// Binary search: find largest index entry ≤ target
val idx = mmapBuffer.duplicate
val slot = indexSlotFor(targetOffset)
if (slot==-1) return OffsetPosition(base,0)
OffsetPosition(relativeOffset(slot),
physicalPosition(slot))
}
// Index entry fixed 8 bytes: 4-byte relative offset
// + 4-byte physical offset (int can't hold >2GB
// segment-internal offset, also why segment can't be unlimited)Note two details in comments: offset stored as 4-byte relative to segment base , physical offset also 4 bytes — this caps single-segment physical offset at 2 GB, a theoretical constraint on max segment size.
3.2 FileRecords.searchForOffset: Scan Log After Index
// FileRecords.scala (simplified)
def searchForOffset(target: Long, start: Int,
maxPos: Int): SearchResult = {
val buffer = readInto(start, maxPosition)
var pos = start
while (pos < maxPosition) {
val batch = Batch.parse(buffer, pos)
// batch first offset exceeds target: hit
if (batch.baseOffset > target)
return SearchResult(pos, batch.size)
if (batch.lastOffset >= target)
return SearchResult(pos, batch.size)
pos += batch.size
}
// Not found: return -1 so upper layer tries next segment
SearchResult(-1, 0)
}Scan unit is batch, not single message — returns "physical start of batch containing target offset"; consumer takes whole batch and picks. That's why consume request's minimum unit is also batch: one index lookup + one batch read = shortest consume read path.
3.3 deleteOldSegments: Three-Dimension Deletion
// UnifiedLog.scala (simplified)
def deleteOldSegments(): Unit = {
// Three dimensions each judge, take union
deleteRetentionMsBreachedSegments()
deleteRetentionSizeBreachedSegments()
deleteLogStartOffsetBreachedSegments()
}
// Judgment logic: segment's latest timestamp vs retention
// Timestamp source: message-carried or broker-written
// (log.message.timestamp.type config)
// Delete action: first advance log start offset, then async unlink
// (Linux: open file can still be read to end)The third dimension logStartOffset is noteworthy: not just retention's product, compaction and manual alter also move it . Consumer offset below it triggers OffsetOutOfRange — many "mysterious offset out of range" trace back to someone changing retention causing log start offset to jump forward.
3.4 LogCleaner: Compaction Merge Process
// LogCleaner.scala (simplified)
// 1. Pick segments: group with dirty ratio > min.cleanable
// 2. Load offset map (dedupe buffer,
// key hash -> latest position, if over limit
// drop old entries with 1/16 probability to avoid OOM)
// 3. Rewrite: read dirty segments record by record,
// key not in map or map value == current -> write to clean segment
// 4. Tombstone handling: tombstone retained
// delete.retention.ms then purged
// 5. Atomic rename: clean segment swapped in,
// failure discards and retries (idempotent)Step 2's "over-limit drop old entries" explains a phenomenon: cleaner thread memory spikes during large-topic compaction — dedupe buffer defaults proportional to segment.bytes; topics with huge key counts/lengths must either increase log.cleaner.dedupe.buffer.size or accept "incomplete cleaning."
Advanced: Seven Accounts
4.1 Capacity Account: Reserve for Worst Case
Capacity = throughput × retention × replicas ÷ disks × 1.3, plus "downstream full stop one day" rebound. Monitor water levels at 70/85. Biggest estimation error: compression ratio . Upstream zstd compresses 3:1, disk sees 1 GB/hr; if upstream disables compression one day, disk instantly 3x — capacity must plan on uncompressed worst case , not current observed values.
4.2 Index Account: Density Is Read/Write Contract
Smaller index.interval.bytes = denser index: shorter query scans, faster random reads, but larger index files, higher write-path maintenance cost. Consume scenarios (sequential tail reads) barely use index — can relax to 8 KB or 16 KB to save memory; heavy random replay (risk replay, audit query) keep 4 KB. This account's essence: index density is insurance bought for "random reads" — no random reads, don't buy.
4.3 Flush Account: Defaults Off Is Optimal
Only legitimate case to enable flush: single-replica topics (e.g., pure cache dump topics). Under multi-replica + acks=all, keep all flush params default, hand durability to replica mechanism. Worried about "page cache loss"? Re-read Section 1.5's conclusion: reliability comes from replicas, not disk .
4.4 Compact Account: It's Not a Table, Don't Use It as One
Correct compact usage is "changelog": upstream sends change events with stable keys , downstream reads converged state. Incident 3's anti-pattern (full dump + timestamp key) treats it as a table. Three disciplines: keys stable and unique (business primary key, no timestamps); deletes must emit tombstones, not rely on "next full dump overwrites"; pair with delete.retention.ms to give downstream delete-visibility window. One hidden trap: compact topics have no "expiration" concept — retention.ms only works if cleanup policy also includes delete; pure compact topics only control size via key convergence.
4.5 Migration Account: Reassignment Is Storage Heavy Lifting
kafka-reassign-partitionscopies entire partition history from source to target broker — migrating a 90 GB partition = 90 GB network + disk I/O. Large clusters must throttle ( replication.quota.window.num configs) and batch; moving dozens of large partitions at once evicts page cache, saturates replication bandwidth, slows normal follower sync. Migration window: off-peak, batch by topic, watch bandwidth and lag .
4.6 Disk Account: JBOD, RAID, Cloud Disks
Physical machine era standard: JBOD — multiple disks direct-mounted, log.dirs comma-listed, Kafka spreads by partition — one disk fails, only its partitions lost (other replicas survive), failed disk's partition leaders fail over. RAID10 survives disk loss but requires full rebuild, double cost. Cloud era changed: cloud disks are already multi-replica storage; RAID pointless, JBOD unnecessary — just single large disk . Failure model shifts from "disk dies" to "host dies"; broker multi-replica covers it. Multi-disk log.dirs on cloud disks introduces "single disk fills" scheduling issues (Kafka spreads by partition count, not capacity); cloud env single-disk single-directory simpler.
4.7 Version Account: Three Storage-Related Changes
0.11 introduced message format v2 (batch-level compression instead of per-message, big compression ratio & producer throughput gains) and idempotent transactions; old consumer upgrades had massive compatibility issues.
2.x defaults to v2; new clusters without legacy baggage need not care.
KIP-405 (Tiered Storage) production-ready in 3.6: cold segments move to object storage (S3/OSS), broker keeps only hot data; retention extends to months without local disk growth — rewrites the "capacity account" formula, structural win for long-retention log scenarios (compliance audit), but adds object storage dependency (availability & billing); extra failure domain. Adopt only after evaluating cold-data read frequency — write-once-read-never archival benefits most.
High-Frequency Interview Questions (with Answers)
The article includes 20 detailed Q&A pairs covering: why append-only vs B-tree, sparse index mechanics, .index vs .timeindex, power loss & fsync, timestamp field dangers, compaction & tombstones, segment size limits, page cache & JVM sizing, retention misconfiguration causes, compact "missing" messages, slow broker restart causes, multi-disk log.dirs allocation, OffsetOutOfRange handling, partition migration storage pressure, zero-copy & TLS interaction, log.segment.ms vs log.segment.bytes tuning, compact dirty ratio tuning, tiered storage trade-offs, storage metrics to monitor, and capacity review checklist. Each answer is grounded in the article's technical analysis.
One-Page Cheat Sheet
Storage Five-Piece: Partition = directory; Segment = log + index + timeindex trio; Filename = base offset; Only active segment writable; Deletion granularity = whole segment.
Four Formulas: Capacity = throughput × retention × replicas ÷ disks × 1.3; Lookup = binary search index + scan one interval; Recovery time ≈ data volume ÷ sequential read; Water alerts = 70% / 85%.
Three Iron Laws: Reliability from replicas not flush; Low-throughput topics must set segment.ms; Compact is not a table — keys stable, deletes via tombstones.
Three Monitors: Disk water dual thresholds, broker load time (10 min), cleaner backlog. With these, Incidents 1 & 2 would be visible a week before explosion.
Summary & Next Episode Preview
All storage layer design revolves around one judgment: message logs are immutable . Immutable → append-only beats B-tree; immutable → sparse index suffices; immutable → page cache + zero-copy own the read path; and because immutable, "deletion" becomes an independent subsystem (delete & compact), with all pitfalls concentrated there. Remember three incident lessons: recovery speed scales with segment size (keep segments small); retention checks config hierarchy (topic overrides broker); compact isn't a table (keys must be stable). Next episode enters the heart of replication: ISR & High Watermark — what does acks=all actually wait for? How long is the ISR shrink detection window? How to decide on unclean.leader.election? We'll start with a "ISR from 3 down to 1 in 12 minutes" case.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
