Nov 3, 2022 · Big Data

Why Kafka Stores Data the Way It Does: A Deep Dive into Its Log Architecture

This article thoroughly examines Kafka's storage system, explaining why it uses sequential log writes combined with sparse indexing, how different log formats evolved, and the mechanisms for log retention and compaction that enable high‑throughput, fault‑tolerant streaming at massive scale.

Big DataKafkaLog Compaction

0 likes · 22 min read

Why Kafka Stores Data the Way It Does: A Deep Dive into Its Log Architecture

Big Data Technology Architecture

Oct 15, 2021 · Backend Development

Key Concepts and Internal Mechanisms of Apache Kafka

This article explains Kafka's internal topics, preferred replicas, partition assignment processes, log directory layout, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, legacy consumer design flaws, rebalance workflow, and producer idempotence, providing a comprehensive overview of Kafka's backend architecture.

Consumer OffsetsLog RetentionPartition Assignment

0 likes · 16 min read

Key Concepts and Internal Mechanisms of Apache Kafka