Comparison of Kafka and Pulsar Stream Consumption Models and Rebalance Mechanisms

Kafka uses a stream consumption model with consumer groups and dynamic rebalance when partitions or group membership changes, allowing consumers to be reassigned without restarting.

Pulsar abstracts both queue and stream models, offering subscription types such as Exclusive, Failover, Share, and the newer Key_Shared, and supports similar partition assignment.

In a comparison, Kafka’s rebalance distributes partitions among consumers in a group, while Pulsar’s Failover subscription uses a hash‑based assignment and promotes a leader consumer, with sub‑consumers acting as followers.

Example Java code shows how to create two Pulsar consumers with the same subscription name and Failover type:

Consumer<byte[]> consumer1 = pulsarClient.newConsumer()
    .topic("topic-1")
    .subscriptionName("my-subscriber-name")
    .subscriptionType(SubscriptionType.Failover)
    .subscribe();

Consumer<byte[]> consumer2 = pulsarClient.newConsumer()
    .topic("topic-2")
    .subscriptionName("my-subscriber-name")
    .subscriptionType(SubscriptionType.Failover)
    .subscribe();

Scenario 1 demonstrates running Pulsar in standalone mode, creating a 4‑partition topic, starting three Failover consumers in the same subscription, producing ten messages, and observing the expected distribution (2, 5, 3 messages).

Scenario 2 shows two Exclusive consumers on the same subscription, where the second consumer fails with “Exclusive consumer is already connected”, illustrating the exclusivity rule.

Overall, the article highlights how Kafka’s rebalance and Pulsar’s subscription mechanisms achieve dynamic load balancing and fault tolerance in stream processing.