Kafka vs Pulsar: Choosing the Right Messaging System for Architecture and Production

Abstract Kafka and Pulsar are mature distributed messaging systems that address different resource‑model constraints. Kafka achieves extreme throughput and low latency with a short write path that couples compute and storage. Pulsar separates brokers from BookKeeper storage, enabling elastic broker scaling, multi‑tenant isolation, and massive Topic management. Selecting between them requires matching the workload bottleneck (throughput, scaling, Topic count, retention, failure recovery) and the team’s operational capability.

Why teams oscillate between Kafka and Pulsar

Typical conclusions – Kafka offers high throughput, a large ecosystem, and strong community support; Pulsar offers an advanced architecture, strong scalability, and multi‑tenant friendliness – are correct but insufficient for production decisions. Real‑world pain points include:

Control‑plane stability when Topic count grows from dozens to thousands.

Cold‑data reads degrading hot‑write latency as retention expands from 1 day to 30 days.

Consumer rebalance storms during frequent scaling or rolling upgrades.

Resource isolation for multi‑tenant platforms.

Mean‑time‑to‑recovery (minutes vs. hours) after cluster failures.

Quick decision guide

Scenarios favoring Kafka

Extreme throughput and sub‑millisecond end‑to‑end latency are required.

Topic count is moderate (tens to a few hundred) and partition management is mature.

Primary use cases are log collection, event tracking, CDC, or streaming‑compute buses.

Team already has deep Kafka operational experience.

Heavy reliance on Kafka Connect, Schema Registry, Flink/Spark ecosystems.

Scenarios favoring Pulsar

Massive Topic count, clear multi‑tenant boundaries, and complex business domains.

Need for compute‑storage separation and frequent broker horizontal scaling.

Long‑term retention with hot‑cold data tiering.

Multiple subscription models (real‑time, replay, audit) on the same data.

Fine‑grained namespace throttling, quota, and isolation requirements.

When neither is needed immediately

Daily message volume stays below a few million.

No clear streaming, asynchronous decoupling, or peak‑shaving needs.

Team lacks sustainable operational capacity.

Introducing complex infrastructure solely for “advanced architecture” is unjustified.

Fundamental architectural differences

Kafka: Partition‑log‑centric tight coupling

Core abstraction is a Partition – an ordered append‑only log with a leader handling writes and followers replicating. A broker handles client requests, replica management, and local log persistence, binding compute and storage.

Short write path, sequential I/O, high page‑cache hit rate → excellent throughput and latency.

Scaling requires partition rebalancing; metadata grows with partition count; hot writes and cold reads compete for the same disks.

Pulsar: Layered service with compute‑storage decoupling

Broker handles ingress, routing, caching, and protocol; BookKeeper stores messages in ledgers; ZooKeeper (or equivalent) coordinates metadata. Producers send to a broker, which writes to a BookKeeper ensemble; the broker remains largely stateless.

Broker can scale horizontally without moving data.

Topic growth pressure shifts to metadata/storage layers, not local log files.

Long‑term retention benefits from tiered storage and hot‑cold separation.

Extra hop adds latency; failure diagnosis spans three layers; BookKeeper tuning has a higher barrier.

Performance gap from architectural roots

Write‑path comparison

Kafka: Producer → Partition leader → local log → follower replication → ack.

Pulsar: Producer → Broker → BookKeeper ensemble write → quorum ack → Broker returns success.

Kafka’s short path yields lower latency; Pulsar’s extra hop provides storage scalability and flexible resource management.

Read path & hot‑cold separation

Kafka shares the same disks for hot writes and cold reads, so large back‑fills can affect hot paths. Pulsar’s broker‑storage split allows cold data to be tiered to object storage, making it suitable for long retention, replay, and event‑bus workloads.

Partition & Topic governance pressure

Kafka scales by increasing partitions; more partitions increase control‑plane metadata, rebalance complexity, and per‑broker file‑handle pressure. Pulsar decouples Topic management from ledger storage, handling thousands to millions of Topics more gracefully, especially with per‑tenant namespace governance.

Five practical decision dimensions

Throughput & latency targets – Kafka typically excels for ultra‑high throughput and low P99 latency.

Topic & partition scale – Kafka fits tens to hundreds of Topics; Pulsar shines with thousands to millions.

Message retention strategy – Short‑term buffering favors Kafka; long‑term, replayable assets favor Pulsar.

Failure recovery & ops model – Kafka’s complexity lies in partition management and ISR; Pulsar’s complexity lies in BookKeeper tuning and multi‑layer fault paths.

Organizational capability & ecosystem fit – Teams deep in Kafka Connect, Debezium, Flink/Spark benefit from Kafka; teams building a unified multi‑tenant event platform benefit from Pulsar.

Architect’s decision framework

Core demand: extreme throughput → Kafka; elastic governance → Pulsar.

Topic/tenant scale: moderate → Kafka; massive → Pulsar.

Message nature: short‑term pipe → Kafka; long‑term event asset → Pulsar.

Team skill: single‑layer broker ops → Kafka; layered infra ops → Pulsar.

Dependency on ready‑made connectors: heavy → Kafka; light → Pulsar.

Frequency of broker horizontal scaling: infrequent → Kafka; frequent → Pulsar.

Production‑grade architecture example (real‑time risk control platform)

Client/Service
  |
  v
API Gateway / gRPC Gateway
  |
  v
Event Ingress Service
  |-- Schema Registry / contract validation
  |-- Idempotent key generation / trace injection / rate limiting
  v
Message Bus (Kafka or Pulsar)
  |-- Real‑time rule engine
  |-- Flink stream processing
  |-- Risk profile updates
  |-- Offline data lake ingestion
  |-- Dead‑letter & compensation handling

Key engineering capabilities that determine system availability:

Message contract governance

Idempotency and deduplication

Back‑pressure and rate limiting

Consumer failure retries

Dead‑letter isolation

Observability and capacity planning

Recommended event model

{
  "eventId": "01JX8Y1F72H8D3R4M2J4N7W9AB",
  "eventType": "risk.trade.created",
  "source": "trade-service",
  "tenantId": "tenant-a",
  "aggregateId": "order-102938",
  "traceId": "7b0f5d2f6d8b4e2a",
  "occurredAt": 1747889654123,
  "partitionKey": "user-8891",
  "schemaVersion": 3,
  "payload": {
    "userId": "user-8891",
    "orderId": "order-102938",
    "amount": 250000,
    "currency": "CNY",
    "deviceId": "ios-238712"
  }
}

Field importance: eventId – idempotent deduplication and tracing. eventType – routing and version governance. tenantId – required for multi‑tenant isolation. aggregateId – enables event sourcing and entity‑level replay. traceId – links distributed traces. partitionKey – guarantees ordering within a partition. schemaVersion – supports backward‑compatible evolution.

Production‑grade producer (Java + Spring Boot)

Domain event record

package com.example.messaging.domain;

import java.time.Instant;
import java.util.Map;

public record RiskEvent(
    String eventId,
    String eventType,
    String tenantId,
    String aggregateId,
    String partitionKey,
    String traceId,
    Instant occurredAt,
    int schemaVersion,
    Map<String, Object> payload
) { }

Send result model

package com.example.messaging.producer;

public record SendResult(
    String messageId,
    String topic,
    int partition,
    long offset
) { }

Event headers builder

package com.example.messaging.producer;

import com.example.messaging.domain.RiskEvent;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.Map;
import org.springframework.stereotype.Component;

@Component
public class EventHeadersBuilder {
    public Map<String, byte[]> build(RiskEvent event) {
        Map<String, byte[]> headers = new LinkedHashMap<>();
        headers.put("eventId", bytes(event.eventId()));
        headers.put("eventType", bytes(event.eventType()));
        headers.put("tenantId", bytes(event.tenantId()));
        headers.put("traceId", bytes(event.traceId()));
        headers.put("schemaVersion", bytes(String.valueOf(event.schemaVersion())));
        return headers;
    }
    private byte[] bytes(String value) {
        return value == null ? new byte[0] : value.getBytes(StandardCharsets.UTF_8);
    }
}

Kafka producer configuration & publishing

package com.example.messaging.producer.kafka;

import com.example.messaging.domain.RiskEvent;
import com.example.messaging.producer.EventHeadersBuilder;
import com.example.messaging.producer.SendResult;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.internals.RecordHeaders;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import org.springframework.stereotype.Service;

@Configuration
public class KafkaProducerConfiguration {
    @Bean
    public ProducerFactory<String, String> producerFactory(@Value("${messaging.kafka.bootstrap-servers}") String bootstrapServers) {
        Map<String, Object> props = Map.ofEntries(
            Map.entry(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers),
            Map.entry(ProducerConfig.ACKS_CONFIG, "all"),
            Map.entry(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true),
            Map.entry(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5),
            Map.entry(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE),
            Map.entry(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"),
            Map.entry(ProducerConfig.LINGER_MS_CONFIG, 5),
            Map.entry(ProducerConfig.BATCH_SIZE_CONFIG, 128 * 1024),
            Map.entry(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000),
            Map.entry(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class),
            Map.entry(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class)
        );
        return new DefaultKafkaProducerFactory<>(props);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate(ProducerFactory<String, String> producerFactory) {
        return new KafkaTemplate<>(producerFactory);
    }
}

@Service
public class KafkaRiskEventPublisher {
    private final KafkaTemplate<String, String> kafkaTemplate;
    private final ObjectMapper objectMapper;
    private final EventHeadersBuilder headersBuilder;
    private final MeterRegistry meterRegistry;

    public KafkaRiskEventPublisher(KafkaTemplate<String, String> kafkaTemplate,
                                   ObjectMapper objectMapper,
                                   EventHeadersBuilder headersBuilder,
                                   MeterRegistry meterRegistry) {
        this.kafkaTemplate = kafkaTemplate;
        this.objectMapper = objectMapper;
        this.headersBuilder = headersBuilder;
        this.meterRegistry = meterRegistry;
    }

    public CompletableFuture<SendResult> publish(String topic, RiskEvent event) {
        Timer.Sample sample = Timer.start(meterRegistry);
        try {
            String body = objectMapper.writeValueAsString(event);
            RecordHeaders headers = new RecordHeaders();
            headersBuilder.build(event).forEach(headers::add);
            ProducerRecord<String, String> record = new ProducerRecord<>(topic, null, event.partitionKey(), body, headers);
            return kafkaTemplate.send(record)
                .thenApply(r -> toResult(topic, sample, r))
                .exceptionally(ex -> {
                    meterRegistry.counter("messaging.producer.failure", "topic", topic).increment();
                    throw new RuntimeException("Kafka publish failed, eventId=" + event.eventId(), ex);
                });
        } catch (Exception ex) {
            sample.stop(meterRegistry.timer("messaging.producer.serialize.error", "topic", topic));
            throw new IllegalStateException("Serialize event failed, eventId=" + event.eventId(), ex);
        }
    }

    private SendResult toResult(String topic, Timer.Sample sample, org.springframework.kafka.support.SendResult<String, String> result) {
        sample.stop(meterRegistry.timer("messaging.producer.latency", "topic", topic));
        meterRegistry.counter("messaging.producer.success", "topic", topic).increment();
        var metadata = result.getRecordMetadata();
        return new SendResult(
            metadata.topic() + "-" + metadata.partition() + "-" + metadata.offset(),
            topic,
            metadata.partition(),
            metadata.offset()
        );
    }
}

Key production‑grade aspects:

Enable acks=all and idempotent sending to reduce duplicates and loss.

Use compression and a small linger to improve small‑message throughput.

Explicit event headers simplify audit and tracing.

Standardised send result eases downstream integration for both Kafka and Pulsar.

Micrometer metrics provide immediate observability.

Production‑grade consumer design

Common mistake: treating message handling as a plain method call. Robust consumption must be built around failure handling.

Layered consumer responsibilities

Listener layer – receives messages and performs protocol conversion.

Application layer – idempotency checks, rate limiting, transaction orchestration.

Domain layer – actual business logic.

Infra layer – database writes, cache updates, downstream calls.

Idempotent consumption guard (Redis‑backed)

package com.example.messaging.consumer;

import java.time.Duration;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

@Component
public class IdempotentMessageGuard {
    private final StringRedisTemplate redisTemplate;
    public IdempotentMessageGuard(StringRedisTemplate redisTemplate) { this.redisTemplate = redisTemplate; }
    public boolean tryAcquire(String eventId) {
        Boolean success = redisTemplate.opsForValue()
            .setIfAbsent("msg:idempotent:" + eventId, "1", Duration.ofHours(24));
        return Boolean.TRUE.equals(success);
    }
}

Kafka listener example

package com.example.messaging.consumer.kafka;

import com.example.messaging.consumer.IdempotentMessageGuard;
import com.example.messaging.domain.RiskEvent;
import com.example.messaging.consumer.RiskEventApplicationService;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.micrometer.core.instrument.MeterRegistry;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Component;

@Component
public class RiskEventKafkaListener {
    private final ObjectMapper objectMapper;
    private final RiskEventApplicationService applicationService;
    private final IdempotentMessageGuard idempotentGuard;
    private final MeterRegistry meterRegistry;

    public RiskEventKafkaListener(ObjectMapper objectMapper,
                                 RiskEventApplicationService applicationService,
                                 IdempotentMessageGuard idempotentGuard,
                                 MeterRegistry meterRegistry) {
        this.objectMapper = objectMapper;
        this.applicationService = applicationService;
        this.idempotentGuard = idempotentGuard;
        this.meterRegistry = meterRegistry;
    }

    @KafkaListener(topics = "${messaging.kafka.topics.risk}",
                   groupId = "${messaging.kafka.groups.risk-processor}",
                   containerFactory = "riskKafkaListenerContainerFactory")
    public void onMessage(ConsumerRecord<String, String> record, Acknowledgment ack) {
        try {
            RiskEvent event = objectMapper.readValue(record.value(), RiskEvent.class);
            if (!idempotentGuard.tryAcquire(event.eventId())) {
                ack.acknowledge();
                return;
            }
            applicationService.handle(event);
            ack.acknowledge();
            meterRegistry.counter("messaging.consumer.success", "topic", record.topic()).increment();
        } catch (RetryableBusinessException ex) {
            meterRegistry.counter("messaging.consumer.retry", "topic", record.topic()).increment();
            throw ex;
        } catch (Exception ex) {
            // dead‑letter handling omitted for brevity
            ack.acknowledge();
            meterRegistry.counter("messaging.consumer.dlq", "topic", record.topic()).increment();
        }
    }
}

Consumer best practices:

Manual offset commits – avoid default auto‑commit.

Separate retryable and non‑retryable exceptions.

Dead‑letter queue must retain original payload and failure reason.

Idempotency should not rely solely on DB unique keys.

Back‑pressure strategies are required when downstream is slow.

Engineering concerns specific to Pulsar

BookKeeper principles

Journal must use low‑latency SSD/NVMe.

Ledger can reside on larger‑capacity disks. ensemble / write quorum / ack quorum must match availability goals.

Recovery paths need proactive drills, not just static config.

Key Pulsar capabilities to leverage

Namespace‑level throttling, quota, and isolation.

Tiered storage reduces hot‑storage cost.

Multi‑subscription model supports real‑time, replay, and audit consumption.

Stateless broker scaling must be coordinated with connection governance.

Common misconceptions

Assuming broker scaling is trivial while ignoring BookKeeper’s role.

Believing massive Topic count automatically mandates Pulsar without considering team’s fault‑diagnosis expertise.

Thinking storage separation always cuts cost; poorly designed hot‑cold tiering can increase expense.

Capacity planning for high‑concurrency scenarios

Write‑throughput formula

Write bandwidth = messages‑per‑second × avg‑message‑size × replica‑factor × amplification

Amplification includes protocol headers, batch overhead, compression delta, and log‑segment metadata.

Storage capacity formula

Total storage = messages‑per‑second × avg‑message‑size × retention‑days × replica‑factor × redundancy

Guidelines: keep disk utilization at 65‑75 %, CPU peak below 60 %, and reserve network headroom for rebalancing and failure recovery.

Risk‑control case study

Assumptions: 1.5 M msgs/s, 1.2 KB avg size, replica factor 3, 7‑day retention. 1,500,000 × 1.2 KB × 3 ≈ 5.4 GB/s Without compression, batching, or traffic shaping, such scale stresses disk, network, page cache, and downstream consumers continuously.

Fault‑handling differences

Kafka typical issues

Rebalance storms.

Partition skew.

Broker disk exhaustion.

ISR shrinkage limiting writes.

Large replay batches throttling hot path.

Pulsar typical issues

BookKeeper journal jitter.

Long ledger recovery times.

Uneven broker distribution after stateless scaling.

Backlog buildup increasing latency.

Metadata service instability affecting control plane.

Essential observability metrics

Producer success/failure/retry rates.

Broker inbound/outbound traffic and request queue length.

Topic/partition growth trends.

Consumer lag/backlog.

Disk usage, I/O wait, page‑cache hit ratio.

P95/P99 latency.

Dead‑letter and retry message volumes.

Kubernetes & cloud‑native deployment guidance

Kafka on Kubernetes

Deploy brokers as a StatefulSet.

Mount dedicated data disks, separate from system disks.

Avoid using rebalancing as a routine scaling method.

Configure node affinity/anti‑affinity.

Before rolling upgrades, assess consumer static members and ISR health.

Pulsar on Kubernetes

Separate deployment of brokers, BookKeeper, and metadata components.

Use distinct storage classes for journal (SSD) and ledger (HDD).

Scale only brokers elastically; avoid frequent BookKeeper scaling.

Define PodDisruptionBudget to protect against excessive replica churn.

Shared principles

Network quality is critical; message systems are highly sensitive to jitter.

Ensure monitoring and log collection do not compete with data‑disk I/O.

Load tests must reflect realistic message size distribution, not just a 1 KB baseline.

Real‑world recommendation matrix

Scenario 1 – Log collection & real‑time tracking

Recommended: Kafka – high throughput, mature ecosystem, short integration path to Flink/Spark/ClickHouse.

Scenario 2 – Database CDC & data bus

Recommended: Kafka – Debezium and Kafka Connect maturity, strong transactional guarantees.

Scenario 3 – Multi‑tenant SaaS messaging platform

Recommended: Pulsar – stronger namespace isolation, easier governance of massive Topic counts, platform‑level scaling.

Scenario 4 – IoT massive device ingestion

Recommended: Pulsar – huge per‑device Topic count, long‑lived connections, extensive replay needs.

Scenario 5 – Trading risk & real‑time decision chain

Ultra‑low latency → Kafka.

Multi‑tenant, long retention, flexible replay → Pulsar.

Anti‑patterns to avoid

Relying on open‑source benchmarks as the sole selection basis.

Treating a message system as a replacement for a database.

Focusing only on broker design while ignoring consumer reliability.

Separating retention policy from capacity planning.

Introducing heavyweight infrastructure before the business scale justifies it.

Evolution roadmap

Stage 1 – Early business

Adopt Kafka for rapid start‑up.

Establish event contracts, idempotency, dead‑letter handling, and monitoring.

Avoid over‑architecting a large platform initially.

Stage 2 – Growth phase

Evaluate Topic count, retention, replay needs, and consumer complexity.

If Kafka shows partition‑governance pressure, consider multi‑cluster or migration to Pulsar.

Stage 3 – Platformization

When the organization reaches multi‑tenant, long‑retention, cross‑region replication, Pulsar becomes a more suitable long‑term foundation.

Final verdict

The trade‑off is not about which system is more advanced; it is about which resource model aligns with the workload.

Choose Kafka for extreme throughput, low latency, mature ecosystem, and straightforward operations.

Choose Pulsar for massive Topic/tenant scale, compute‑storage separation, flexible scaling, and long‑term retention.

A professional architect asks where the current bottleneck lies, how it will evolve, whether the team can operate the system, and how fault recovery will be handled. Answering these questions makes the Kafka vs Pulsar decision clear.

Selection checklist (ready‑to‑use during architecture review)

Peak messages per second, average and max payload size.

Retention period and historical replay requirements.

Projected three‑month growth of Topics and partitions.

Existence of multi‑tenant isolation demands.

Completeness of retry, dead‑letter, and compensation flows after consumer failures.

Unified schema governance and versioning strategy.

Sufficient monitoring, load‑test data, and fault‑drill records.

Actual operational experience of the team with Kafka or Pulsar.

If any of these items lack answers, define system constraints before debating the technology.