Why Kafka Dominates Distributed Messaging: Architecture, Guarantees, and Best Practices

Abstract

Kafka, originally developed by LinkedIn and now an Apache project, is a distributed publish/subscribe messaging system known for high throughput and scalability. The article covers Kafka's background, design objectives, advantages of using a message system, a comparison with other queues, and detailed architecture.

Background

Kafka was created to handle LinkedIn's activity stream and operational data pipelines, replacing log‑file based batch processing with a more robust infrastructure for real‑time and offline data handling.

Design Goals

O(1) time complexity for message persistence, even at terabyte scale.

High throughput (100K+ messages/second on commodity hardware).

Partitioned messaging with ordered delivery per partition.

Support for both offline and real‑time processing.

Horizontal scalability (scale‑out).

Why Use a Message System

Decoupling : Provides an interface layer that isolates producers and consumers.

Redundancy : Persists messages to avoid data loss.

Scalability : Easy to increase throughput by adding producers/consumers.

Flexibility & Peak Handling : Buffers bursts of traffic without over‑provisioning.

Recoverability : System components can fail without stopping the whole pipeline.

Order Guarantees : Preserves message order within a partition.

Buffering : Improves overall processing efficiency.

Asynchronous Communication : Allows producers to enqueue messages without waiting for immediate processing.

Message‑Queue Comparison

RabbitMQ : Erlang‑based, heavyweight, broker‑centric, strong routing and persistence.

Redis : Key‑value store with lightweight queue capabilities; excels with small messages.

ZeroMQ : Fast, broker‑less, but lacks persistence.

ActiveMQ : Apache project, similar to ZeroMQ with broker support.

Kafka/Jafka : High‑performance, O(1) persistence, horizontal scaling, integrates with Hadoop and Spark.

Kafka Architecture

Terminology

Broker : A server in a Kafka cluster.

Topic : Logical category of messages.

Partition : Physical subdivision of a topic.

Producer : Publishes messages to brokers.

Consumer : Reads messages from brokers.

Consumer Group : Set of consumers sharing a group ID.

Topology

A typical cluster contains multiple producers, brokers, consumer groups, and a Zookeeper ensemble that manages configuration, leader election, and rebalancing.

Topic & Partition

Topics are logical queues; partitions are physical log segments stored as files. Each message receives a 64‑bit offset within its partition. Log entries consist of a length, magic byte, CRC, and payload.

Segments are named by the first offset and have accompanying index files.

Retention Policies

Kafka can delete old data based on time or segment size. Example configuration:

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168
# Maximum size of a log segment file
log.segment.bytes=1073741824
# Interval to check log segments for deletion
log.retention.check.interval.ms=300000
# Enable log cleaner for compaction
log.cleaner.enable=true

Producer Message Routing

Producers assign messages to partitions using a key. The default partitioner can be overridden. Example custom partitioner:

import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

public class JasonPartitioner<T> implements Partitioner {
    public JasonPartitioner(VerifiableProperties verifiableProperties) {}
    @Override
    public int partition(Object key, int numPartitions) {
        try {
            int partitionNum = Integer.parseInt((String) key);
            return Math.abs(Integer.parseInt((String) key) % numPartitions);
        } catch (Exception e) {
            return Math.abs(key.hashCode() % numPartitions);
        }
    }
}

Sending messages with this partitioner ensures that messages sharing the same key land in the same partition.

public void sendMessage() throws InterruptedException {
    for (int i = 1; i <= 5; i++) {
        List<KeyedMessage<String, String>> messageList = new ArrayList<>();
        for (int j = 0; j < 4; j++) {
            messageList.add(new KeyedMessage<String, String>("topic2", String.valueOf(j),
                String.format("The %d message for key %d", i, j)));
        }
        producer.send(messageList);
    }
    producer.close();
}

Consumer Group

Within a consumer group, each partition is consumed by only one consumer, while multiple groups can read the same messages, enabling both broadcast and unicast semantics.

Push vs. Pull

Kafka uses a push model for producers and a pull model for consumers. Pull allows consumers to control their own consumption rate, which is better suited for handling variable workloads.

Delivery Guarantees

At most once

: Messages may be lost but never duplicated. At least once: Messages are never lost but may be duplicated. Exactly once: Each message is delivered once and only once; requires coordination with external storage and is not fully supported in older Kafka versions.

By default, Kafka guarantees At least once for producer‑to‑broker delivery and can be configured for At most once via asynchronous sends. Achieving true exactly‑once semantics involves two‑phase commits and careful offset management.