In‑Depth Analysis of Apache Kafka: Architecture, Core Concepts, and Benchmark

Background Introduction

This article is originally from Jason’s Blog and analyzes Kafka in depth.

Kafka Overview

Kafka is a distributed, publish/subscribe messaging system designed to provide:

O(1) time persistence even for terabytes of data.

High throughput (up to 100K messages per second on cheap hardware).

Partitioned messaging across Kafka servers with ordered delivery per partition.

Support for both offline and real‑time data processing.

Why Use a Message System

Decoupling : Provides an implicit data‑driven interface that allows independent evolution of producers and consumers.

Redundancy : Persists messages until they are fully processed, preventing data loss.

Scalability : Adding more producers or consumers does not require code changes.

Flexibility & Peak Handling : Buffers bursts of traffic without over‑provisioning resources.

Recoverability : Failure of a component does not bring down the whole system.

Delivery Guarantees : Ensures messages are actually processed.

Order Guarantees : FIFO processing per partition.

Buffering : Allows fast writes to the queue while downstream processing may be slower.

Data‑flow Insight : Message rates help identify bottlenecks.

Asynchronous Communication : Producers can enqueue messages without waiting for immediate processing.

Comparison of Common Message Queues

RabbitMQ : Erlang‑based, supports many protocols (AMQP, XMPP, etc.), heavyweight, suitable for enterprise use, uses a broker architecture.

Redis : Key‑value NoSQL store that also offers lightweight queue capabilities; excels at small payloads but slower for large messages.

ZeroMQ : Very fast, broker‑less library; requires developers to assemble components and does not provide persistence.

ActiveMQ : Apache project offering both broker and peer‑to‑peer modes.

Kafka / Jafka : High‑performance, distributed publish/subscribe system with O(1) persistence, high throughput, native partitioning, and tight integration with Hadoop for both batch and real‑time processing.

Kafka Analysis

Terminology

Broker : A server in a Kafka cluster.

Topic : Logical category of messages; physically stored separately per broker.

Partition : Physical subdivision of a topic; each partition is a folder containing log and index files.

Producer : Publishes messages to brokers.

Consumer : Reads messages; belongs to a consumer group.

Kafka Architecture

A typical Kafka cluster consists of multiple producers, brokers, consumer groups, and a ZooKeeper ensemble that manages configuration, leader election, and rebalancing. Producers push messages to brokers; consumers pull messages from brokers.

Push vs. Pull

Kafka uses a push model for producers and a pull model for consumers. Push can overwhelm slow consumers, while pull allows consumers to fetch at their own pace.

Topic & Partition

Topics are logical queues; each topic is split into one or more partitions to enable horizontal scaling. Each partition stores messages in sequential log files.

Each log entry consists of:

log entry = [4‑byte length][1‑byte magic][4‑byte CRC][payload]

Segments are named by the offset of their first message and end with .kafka. An index file maps offsets to segment ranges.

Because messages are appended sequentially, disk writes are highly efficient.

Replication & Leader Election

Kafka supports per‑partition replication (configurable in $KAFKA_HOME/config/server.properties). Each partition has a single leader handling all reads/writes; followers replicate the leader’s log. The in‑sync replica (ISR) set contains followers that are up‑to‑date and are eligible for leader election.

Commit semantics depend on the ISR: a message is considered committed only after all ISR members have replicated it. Producers can control acknowledgment behavior via request.required.acks.

Consumer Group

Every consumer instance belongs to a consumer group. Within a group, each partition is consumed by only one consumer, guaranteeing ordered processing per partition. Different groups can consume the same topic independently, enabling both real‑time and batch processing pipelines.

Consumer Rebalance

When the number of consumers or partitions changes, Kafka reassigns partitions using a deterministic algorithm:

Sort all partitions and all consumers.

Assign roughly size(PT) / size(CG) partitions to each consumer based on its index.

Update the partition‑owner registry.

Rebalance is triggered by any consumer or broker join/leave and is coordinated via ZooKeeper.

Message Delivery Guarantees

At most once

: Messages may be lost but never duplicated. At least once: Messages are never lost but may be delivered multiple times. Exactly once: Each message is delivered once and only once (requires additional coordination, e.g., two‑phase commit).

By default, Kafka provides at least once semantics for producers and consumers; at most once can be achieved by asynchronous producer sends, while true exactly once requires application‑level handling.

Benchmark

The author attempted to reproduce performance tests for Kafka 0.8.1 but the original report was lost; however, benchmark results from Kafka co‑creator Jay Kreps are referenced.

— END —