Why Message Queues Are Needed and an Introduction to Kafka

When a user places an order online and a delivery person needs to hand over the package, direct interaction can cause delays, resource contention, and scheduling problems; a middle‑man store (the "message middleware") solves these issues by decoupling the sender and receiver.

Benefits of a message‑queue middleware

Decoupling: The sender deposits items at a common location and notifies the receiver, eliminating the need for synchronous coordination.

Asynchrony: The sender can continue other work after handing off the package, improving overall efficiency.

Peak‑shaving: During high‑traffic periods (e.g., a shopping festival), the middleware buffers requests, preventing the receiver from being overwhelmed.

These concepts lead to real‑world services such as smart lockers and delivery stations.

Message‑queue communication models

Two fundamental patterns are used:

Point‑to‑point (P2P): Messages are placed in a queue and consumed by a single consumer; producers push messages, consumers pull them, often via polling.

Publish‑Subscribe (Pub/Sub): Producers publish to a topic, and all subscribed consumers receive a copy; delivery is push‑based, but differing consumer speeds can cause bottlenecks.

Kafka

Kafka is a high‑throughput distributed publish‑subscribe system designed for large‑scale data streams. It offers persistence, replication, and horizontal scalability.

Core terminology

Producer: The data source that writes messages to Kafka.

Broker: An individual Kafka server instance; each broker has a unique ID.

Topic: A logical channel that groups related messages.

Partition: A topic is split into ordered partitions to enable parallelism and scaling.

Replication: Each partition has one leader and multiple followers; followers replicate the leader’s log for fault tolerance.

Message: The payload stored in a log file, identified by an offset.

Consumer & Consumer Group: Applications that read messages; a group’s consumers share the partitions of a topic without overlapping.

Zookeeper: Coordinates broker metadata and leader election.

Data flow in Kafka

Sending data : Producers always write to the leader of a partition; the leader then replicates to followers. The write is sequential, guaranteeing order within a partition. Partition selection follows three rules: explicit partition, key‑based hash, or round‑robin.

Acknowledgment (ACK) levels : 0 (no confirmation), 1 (leader ack), all (all replicas ack) – trade‑off between latency and durability.

Saving data : Messages are appended to log files on disk using a sequential write pattern. Each partition consists of multiple segment files, each with .log, .index, and .timeindex files. Offsets (8‑byte IDs) provide ordered positioning.

Retention policy : Kafka retains data based on time (default 168 hours) or size (default 1 GB); old segments are deleted without affecting read performance.

Consuming data : Consumers pull from the leader of each partition. Within a consumer group, each partition is assigned to a single consumer, ensuring no duplicate processing. Offsets are now stored in the internal __consumer_offsets topic rather than Zookeeper.

Overall, the article walks through why message queues are essential, compares communication patterns, and gives a comprehensive, illustrated introduction to Kafka’s architecture and operational details.