Why Kafka Dominates Distributed Messaging: Architecture, Features, and Best Practices

Background

Kafka was originally developed at LinkedIn to power activity streams and operational data pipelines. It has since been adopted by many companies as a general‑purpose distributed messaging system.

Kafka Overview

Kafka is a distributed publish/subscribe messaging system designed for:

O(1) message persistence even for terabytes of data.

High throughput (100K+ messages per second on modest hardware).

Partitioned topics with ordered delivery per partition.

Support for both offline batch processing and real‑time streaming.

Horizontal scalability.

Why Use a Message System

Decoupling – Provides an interface layer that lets producers and consumers evolve independently.

Redundancy – Persists messages until they are fully processed, preventing data loss.

Scalability – Adding more producers or consumers does not require code changes.

Peak‑handling – Buffers bursts of traffic so critical components are not overwhelmed.

Recoverability – Failure of a single component does not bring down the whole system.

Ordering – Guarantees message order within a partition.

Buffering – Allows fast producers to write to a queue while slower consumers process at their own pace.

Asynchronous communication – Producers can fire‑and‑forget, consumers handle messages when ready.

Message‑Queue Comparison

RabbitMQ – Erlang‑based, supports many protocols, heavyweight, broker‑centric.

Redis – In‑memory key‑value store with optional MQ features; excels with small messages.

ZeroMQ – Fast, broker‑less, requires custom integration; lacks persistence.

ActiveMQ – Apache project, lightweight, supports both broker and peer‑to‑peer modes.

Kafka/Jafka – Apache project, high‑performance, O(1) persistence, horizontal scaling, strong ordering, integrates with Hadoop for batch and real‑time processing.

Kafka Architecture Terminology

Broker – A server in a Kafka cluster.

Topic – Logical category of messages; stored across brokers.

Partition – Physical subdivision of a topic; each partition is an ordered log.

Producer – Publishes messages to brokers.

Consumer – Reads messages from brokers.

Consumer Group – A set of consumers that share the load for a topic.

Kafka Topology

A typical cluster consists of multiple producers (e.g., web page views, server logs), several brokers, consumer groups, and a ZooKeeper ensemble that manages configuration, leader election, and rebalancing.

Topic & Partition Details

Each topic is split into one or more partitions, each backed by a directory of log segment files. Every message receives a 64‑byte offset that identifies its position within the partition.

message length : 4 bytes (value = 1+4+n)
"magic" value : 1 byte
crc           : 4 bytes
payload       : n bytes

Log segments are named by the offset of their first message and end with .kafka. An accompanying index file maps offsets to segment files.

Retention Policies

Kafka can delete old data based on time or segment size. Example server.properties settings:

# Minimum age of a log file to be eligible for deletion
log.retention.hours=168
# Maximum size of a log segment file
log.segment.bytes=1073741824
# Interval for checking retention
log.retention.check.interval.ms=300000
# Enable log compaction
log.cleaner.enable=false

Deletion does not affect read performance because Kafka’s O(1) lookup is independent of file size.

Producer Message Routing

Producers assign messages to partitions using a partitioner. The default uses the message key’s hash; custom partitioners can implement kafka.producer.Partitioner.

import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

public class JasonPartitioner<T> implements Partitioner {
    public JasonPartitioner(VerifiableProperties verifiableProperties) {}
    @Override
    public int partition(Object key, int numPartitions) {
        try {
            int partitionNum = Integer.parseInt((String) key);
            return Math.abs(Integer.parseInt((String) key) % numPartitions);
        } catch (Exception e) {
            return Math.abs(key.hashCode() % numPartitions);
        }
    }
}

Sending messages with keys ensures that records with the same key land in the same partition:

public void sendMessage() throws InterruptedException {
    for (int i = 1; i <= 5; i++) {
        List<KeyedMessage<String, String>> messageList = new ArrayList<>();
        for (int j = 0; j < 4; j++) {
            messageList.add(new KeyedMessage<String, String>("topic2", j+"", "The " + i + " message for key " + j));
        }
        producer.send(messageList);
    }
    producer.close();
}

Consumer Groups

Within a consumer group, each partition is consumed by only one member, but the same topic can be consumed by multiple groups simultaneously, enabling both broadcast and unicast patterns.

Push vs. Pull

Kafka uses a push model for producers and a pull model for consumers. Pull allows consumers to control their own consumption rate, avoiding overload that can occur with push‑only systems.

Delivery Guarantees

At most once – Messages may be lost but never duplicated.

At least once – No loss, possible duplicates.

Exactly once – Each message is processed exactly once; requires external coordination (e.g., two‑phase commit) and is not fully supported in older Kafka versions.

By default Kafka provides at‑least‑once semantics for producers. Consumers can choose when to commit offsets (auto‑commit, manual commit after processing, etc.), influencing the effective guarantee.