Kafka Core Architecture, Principles, Features, and Application Scenarios

Hi, I am mikechen. In this article I share the fundamentals of Kafka, a cornerstone of large‑scale internet architectures and a must‑know middleware for major tech companies.

Kafka Core Architecture

The architecture consists of the following components:

Topic : a logical stream of messages; each message carries a byte payload.

Producer : any entity that publishes messages to a topic.

Broker : a server (or a cluster of servers) that stores the published messages.

Consumer : an entity that subscribes to one or more topics and pulls messages from brokers.

The diagram below shows producers sending data to brokers, brokers holding multiple topics, and consumers retrieving data from brokers.

Kafka Principle Mechanism

We refer to message publishing as the producer and message subscription as the consumer . The intermediate storage layer is the broker . Producers push messages to brokers, while consumers pull messages from brokers, forming a classic publish‑subscribe model.

Multiple brokers cooperate, and producers and consumers are deployed across business logic layers. Zookeeper coordinates requests and routing, enabling a high‑performance distributed messaging system.

Note the push‑pull detail: producers push data to brokers, whereas consumers actively pull data from brokers.

Role of Zookeeper in Kafka

Zookeeper provides essential coordination for Kafka clusters, storing metadata and ensuring high availability. It acts as the distributed coordination framework that ties together production, storage, and consumption, and enables stateless components to establish subscription relationships and load‑balance automatically.

Kafka Features

High throughput & low latency : processes hundreds of thousands of messages per second with millisecond latency; topics can be split into partitions and consumed by consumer groups.

Scalability : supports hot‑expansion of clusters.

Durability & reliability : messages are persisted to disk and replicated to prevent loss.

Fault tolerance : the cluster can tolerate up to n‑1 node failures when replication factor is n.

High concurrency : thousands of clients can read and write simultaneously.

Application Scenarios

Log collection : centralize logs from various services and expose them to consumers such as Hadoop, HBase, or Solr.

Message system : decouple producers and consumers, provide buffering and reliable delivery.

User activity tracking : capture web or app events (clicks, searches, page views) for real‑time monitoring or offline analysis.

Operational metrics : aggregate monitoring data, alarms, and reports across distributed applications.

Stream processing : feed data to frameworks like Spark Streaming or Apache Storm for real‑time computation.

For more in‑depth architectural content, follow the "mikechen的互联网架构" public account and reply with the keyword 【架构】 to receive over 300,000 words of original material.