The Growing Role of Apache Kafka in Modern Big Data Architectures

Author Jun Rao writes about Apache Kafka, with translation by Brian Ling, focusing on high performance, stability, and availability.

In recent years, Apache Kafka’s adoption has surged, with major customers such as Uber, Twitter, Netflix, LinkedIn, Yahoo, Cisco, and Goldman Sachs, offering a highly scalable producer‑consumer system for massive message publishing and real‑time consumption.

Traditional databases, despite adding features like search, streaming, and analytics, face two major drawbacks: they become prohibitively expensive when storing large, diverse datasets (e.g., user behavior, operational metrics, logs) that can be 2‑3 times larger than transactional data, and they grow increasingly complex, making maintenance and new feature development difficult.

To overcome these shortcomings, specialized distributed systems have emerged over the past decade. Built for single objectives, they are simpler, cost‑effective on commodity hardware, often based on open‑source projects, and evolve faster than monolithic databases. Hadoop pioneered this trend with HDFS for distributed storage and MapReduce for batch processing, enabling cheap collection and analysis of large datasets.

Key‑value stores: Cassandra, MongoDB, HBase, etc.

Search engines: Elasticsearch, Solr, etc.

Streaming processors: Storm, Spark Streaming, Samza, etc.

Graph platforms: GraphLab, FlockDB, etc.

Time‑series databases: OpenTSDB, etc.

These purpose‑built systems allow companies to gain deeper insights and create novel applications.

However, integrating data into multiple specialized systems presents challenges. A single dataset often needs to be injected into several systems (e.g., logs for offline analysis and for searchable log records), making separate ingestion pipelines impractical.

Moreover, Hadoop typically stores copies of all data types, which hinders real‑time ingestion required by many other systems—something Hadoop cannot guarantee. This gap is where Kafka excels, offering features such as:

Distributed design for high‑capacity storage on commodity hardware.

Support for multiple subscribers, allowing the same data to be consumed repeatedly.

Disk‑based persistence without performance loss, serving both real‑time and batch consumers.

Built‑in data redundancy ensuring high availability for critical data streams.

Most early adopters integrate several specialized systems and use Kafka as a central data hub, streaming data in real time to various downstream platforms. The diagram below illustrates such a streaming data architecture.

Looking ahead, the industry trend is toward coexistence of multiple specialized systems within the big‑data ecosystem. As more companies adopt real‑time processing, distributed publish‑subscribe platforms like Kafka will play an increasingly critical role, prompting a re‑evaluation of data governance workflows that should begin early in the Kafka ingestion stage.