Why Kafka and Pulsar Lead the Distributed Streaming Landscape

Kafka and Pulsar are two leading distributed streaming platforms.

Apache Kafka

Apache Kafka (Kafka) was developed by LinkedIn and open‑sourced in 2011. It is written in Scala and Java and has become one of the most popular distributed streaming platforms. Kafka follows a publish/subscribe model and offers high throughput, persistence, horizontal scalability, and stream processing capabilities.

Apache Pulsar

Apache Pulsar (Pulsar) was created by Yahoo and open‑sourced in 2016 as a “next‑generation cloud‑native distributed streaming platform.” Pulsar integrates messaging, storage, and lightweight function computing, uses a compute‑storage separation architecture, supports multi‑tenancy, persistent storage, cross‑region replication, strong consistency, high throughput, low latency, and high scalability.

Common Core Features

Both Kafka and Pulsar provide the following basic capabilities:

Message System – Publish/subscribe model where producers send messages to the system and consumers receive them.

Storage System – Ability to store large volumes of data with client‑controlled offsets, enabling historical data retention.

Real‑time Data Pipelines – Connectors (Kafka Connect, Pulsar IO) allow ingestion from sources such as MySQL or MongoDB and expose data to downstream applications.

Stream Processing Applications – Kafka Streams and Pulsar Functions (or integration with Spark, Flink) enable complex transformations, aggregations, and joins on streaming data.

Key Characteristics

High Throughput & Low Latency – Both platforms can handle massive message streams with minimal delay.

Persistence & Consistency – Messages are persisted and replicated to ensure durability and consistency.

Scalability – Data is sharded across broker clusters, allowing horizontal expansion.

Fault Tolerance – Node failures do not disrupt overall system operation.

Fundamental Concepts

Four core concepts appear in both systems: Message, Producer, Consumer, and Topic.

Message – The data entity transmitted through the system.

Producer – The application that publishes messages.

Consumer – The application that subscribes to and processes messages.

Topic – Logical grouping of messages; each topic is independent.

Kafka Specific Concepts

Consumer Group – Logical grouping of consumers that share load.

Broker – Server node that stores and serves messages; multiple brokers form a cluster.

Partition – Subdivision of a topic; each partition is stored on a broker, enabling distributed storage.

Replica, AR, ISR – Replication mechanisms ensuring data safety and consistency.

ACK Mechanism – Producers receive acknowledgments after brokers persist messages; consumers acknowledge successful processing, often by committing offsets.

Pulsar Specific Concepts

Subscription Group – Equivalent to Kafka’s consumer group.

Broker – Compute node that forwards messages to BookKeeper storage.

Bookie – Storage node provided by Apache BookKeeper.

Ledger & Entry – BookKeeper’s log structure; a ledger contains entries.

Tenant & Namespace – Multi‑tenant isolation; namespaces group topics within a tenant.

Cluster – Group of brokers; clusters can replicate data across regions.

ACK Mechanism – Similar to Kafka, with message IDs used for acknowledgments.

Architecture Diagrams