Why Kafka Dominates Distributed Messaging: Architecture, Features, and Comparisons

Background

Kafka was originally developed at LinkedIn to power activity streams and operational data pipelines. It has since been adopted by many companies as a general‑purpose distributed messaging system.

Kafka Overview

Kafka is a horizontally scalable, high‑throughput publish/subscribe messaging system written in Scala. Its main design goals include:

O(1) time complexity for message persistence, even with terabytes of data.

High throughput—over 100 K messages per second on inexpensive commodity hardware.

Partitioned, ordered delivery within each partition.

Support for both offline batch processing and real‑time streaming.

Easy horizontal scaling (scale‑out).

Why Use a Message System

Decoupling : Provides an implicit data‑driven interface layer, allowing independent evolution of producers and consumers.

Redundancy : Persists messages until they are fully processed, preventing data loss.

Scalability : Adding more consumers or producers increases throughput without code changes.

Flexibility & Peak Handling : Handles traffic spikes without over‑provisioning resources.

Recoverability : Failure of a single component does not bring down the whole system.

Ordering Guarantees : Guarantees order within a partition.

Buffering : Acts as a buffer to smooth out processing speed differences.

Asynchronous Communication : Producers can fire‑and‑forget, letting consumers process at their own pace.

Common Message Queue Comparison

RabbitMQ : Erlang‑based, heavyweight, supports many protocols, uses a broker‑centric model.

Redis : In‑memory key‑value store with lightweight MQ capabilities; excels with small messages.

ZeroMQ : Broker‑less, ultra‑fast, but lacks persistence; requires custom integration.

ActiveMQ : Apache project offering both broker and peer‑to‑peer modes.

Kafka/Jafka : High‑performance, O(1) persistence, horizontal scaling, strong ordering, integrates with Hadoop for batch and real‑time workloads.

Kafka Architecture

Terminology

Broker : A server in a Kafka cluster.

Topic : Logical category of messages; stored across brokers.

Partition : Physical subdivision of a topic; each partition is an ordered log.

Producer : Publishes messages to brokers.

Consumer : Reads messages from brokers.

Consumer Group : A set of consumers that share the consumption of a topic’s partitions.

Topology

A typical cluster contains multiple producers, brokers, consumer groups, and a ZooKeeper ensemble that manages configuration, leader election, and rebalancing.

Topic & Partition

Topics behave like logical queues; partitions enable linear scalability. Each partition stores messages in sequential log files, each entry consisting of a 4‑byte length, 1‑byte magic, 4‑byte CRC, and the payload.

message length : 4 bytes (value: 1+4+n)
"magic" value : 1 byte
crc : 4 bytes
payload : n bytes

Log segments are named by the first message’s offset and have accompanying index files.

Kafka retains all messages, but old data can be removed based on time or segment size using configuration such as:

# Minimum age of a log file to be eligible for deletion
log.retention.hours=168
# Maximum size of a log segment file
log.segment.bytes=1073741824
# Interval for checking retention policies
log.retention.check.interval.ms=300000
# Enable log compaction
log.cleaner.enable=false

Producer Message Routing

Producers assign messages to partitions based on a key and a partitioner class. The default partitioner uses the key’s hash; custom partitioners can be implemented.

import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

public class JasonPartitioner<T> implements Partitioner {
    public JasonPartitioner(VerifiableProperties verifiableProperties) {}
    @Override
    public int partition(Object key, int numPartitions) {
        try {
            int partitionNum = Integer.parseInt((String) key);
            return Math.abs(Integer.parseInt((String) key) % numPartitions);
        } catch (Exception e) {
            return Math.abs(key.hashCode() % numPartitions);
        }
    }
}

Sending messages with the same key ensures they land in the same partition:

public void sendMessage() throws InterruptedException {
    for (int i = 1; i <= 5; i++) {
        List<KeyedMessage<String, String>> messageList = new ArrayList<>();
        for (int j = 0; j < 4; j++) {
            messageList.add(new KeyedMessage<String, String>("topic2", j+"", "The " + i + " message for key " + j));
        }
        producer.send(messageList);
    }
    producer.close();
}

Consumer Group

With the high‑level consumer API, a message can be consumed by only one consumer within a group, while multiple groups can read the same message independently.

This mechanism enables both broadcast (each group gets the message) and unicast (all consumers in the same group share the load). It also allows simultaneous offline batch processing (e.g., Hadoop) and real‑time streaming (e.g., Storm) by assigning different consumer groups.

Example: A topic with three partitions, one consumer in group1, and three consumers in group2. Group1 receives all three messages; each consumer in group2 receives one distinct message.

Push vs. Pull

Kafka uses a pull model: consumers request messages at their own pace, which avoids overwhelming slow consumers. Push models (e.g., Facebook Scribe) can cause back‑pressure and service degradation.

Kafka Delivery Guarantees

At most once : Messages may be lost but never duplicated.

At least once : No loss, possible duplicates.

Exactly once : Each message is processed exactly once; Kafka provides at‑least‑once by default, and exactly‑once requires external coordination (e.g., two‑phase commit, idempotent processing).

Producer side: By default, Kafka ensures at‑least‑once. Setting the producer to asynchronous send can achieve at‑most‑once.

Consumer side: Commit semantics determine guarantees. Committing before processing yields at‑most‑once; processing before committing yields at‑least‑once. If processing is idempotent, the effective guarantee can be considered exactly‑once.

In summary, Kafka’s default behavior is at‑least‑once, with configuration options to adjust semantics, while true exactly‑once requires careful integration with downstream systems.