Message Queue Showdown: When to Use Kafka, RabbitMQ, or Pulsar

1. Quick Decision Table

For most scenarios remember: Kafka – best for high‑throughput event streams, log‑style data, CDC, replay‑able pipelines. RabbitMQ – ideal for business command messages, flexible routing, delayed or priority tasks. Pulsar – suited for platform‑level, multi‑tenant, massive topic environments with storage‑compute separation.

2. Why Teams Pick the Wrong MQ

Selection errors usually stem from not classifying messages first. In an event‑driven architecture four categories must be distinguished:

Domain events – e.g., OrderCreated, PaymentSucceeded; focus on business state changes.

Integration events – notifications between subsystems, emphasizing decoupling.

Command messages – explicit actions like “send SMS” or “create fulfillment task”.

Data‑flow events – click‑stream, telemetry, CDC; require high throughput, replay, and analytics.

Most production incidents trace back to mismatching the message type with the wrong platform.

3. Five Core Roles of a Message Queue

Decoupling – producers publish without knowing consumer speed.

Peak shaving – asynchronous queues absorb traffic spikes.

Broadcast – one event triggers many parallel subscribers.

Replay – persistent logs enable back‑fill, audit, and state reconstruction.

Platform governance – unified tenant isolation, quotas, auditing, and observability.

4. Underlying Design of Each MQ

4.1 Kafka – Distributed Commit Log

Kafka’s core is a durable, partitioned, replayable commit log rather than a traditional queue.

Producer appends to partition logs; broker does not track per‑message consumption.

Consumers manage offsets themselves, enabling high throughput and zero‑copy writes.

Strengths: extremely high throughput, built‑in replay, mature ecosystem (Flink, Spark, Debezium).

Limitations: not designed for fine‑grained per‑message ACK, complex routing, or long‑term per‑message retries without extra engineering.

4.2 RabbitMQ – Intelligent Routing Proxy

RabbitMQ’s essence is flexible routing via exchanges rather than storage.

Rich exchange types (direct, topic, fanout, headers) and mature queue semantics (ACK, NACK, TTL, DLX, priority, delayed plugins).

Excels at end‑to‑end business message lifecycles, command‑style workflows, and precise routing.

Weaknesses: high cost under sustained high write + long backlog + slow consumers; disk and memory pressure rise quickly.

4.3 Pulsar – Tiered, Cloud‑Native Messaging

Pulsar separates stateless brokers from BookKeeper‑based storage, offering multi‑tenant namespaces and tiered storage.

Broker statelessness enables rapid scaling; storage tiering supports hot‑cold data.

Strong multi‑tenant isolation, quota enforcement, and namespace management.

Costs: more components (Broker + BookKeeper), higher operational complexity, steep learning curve for small teams.

5. Capability Comparison (Summarised)

Architecture : Kafka – distributed log; RabbitMQ – AMQP broker; Pulsar – storage‑compute separated system.

Best at : Kafka – high‑throughput streams; RabbitMQ – business async routing; Pulsar – platform‑wide, multi‑tenant backbone.

Throughput : Kafka ≫ Pulsar ≈ RabbitMQ (medium‑high).

Ordering : Kafka – per‑partition; RabbitMQ – approximate; Pulsar – per‑key partition ordering.

Replay : Kafka strong, Pulsar strong, RabbitMQ weak.

Routing : RabbitMQ strong, Kafka weak (needs app‑level), Pulsar moderate.

Ops complexity : Kafka medium, RabbitMQ medium, Pulsar high.

6. Practical Selection Method (Eight Questions)

Is the message a command or an event ? Commands → RabbitMQ; events → Kafka/Pulsar.

What consistency level is required? Final consistency suffices for most MQs; strong DB‑MQ atomicity relies on the Outbox pattern.

Is historical replay needed? If yes, choose Kafka or Pulsar.

Will the pipeline experience massive backlog? Event‑flow → Kafka; command‑flow → RabbitMQ.

How complex are routing rules? Fan‑out with per‑consumer logic → RabbitMQ; simple topic routing → Kafka/Pulsar.

Does the team have dedicated middleware ops expertise? Small teams should avoid Pulsar as default.

Is multi‑tenant isolation a requirement? Pulsar provides native support.

Can downstream services tolerate asynchronous compensation? If not, avoid async MQ for critical paths.

7. Five Common Pitfalls

Treating Kafka as a transactional queue without proper acks, idempotent producers, and consumer idempotence.

Using RabbitMQ for massive log ingestion; high write + long backlog drives cost and latency.

Adopting Pulsar solely for “future complexity” without real multi‑tenant or tiered‑storage needs.

Expecting MQ to solve distributed transactions; it only provides async decoupling and eventual consistency.

Deploying message sending without a full governance loop (dead‑letter handling, alerting, compensation).

8. Production‑Grade Design Example

A layered model that combines Outbox, CDC, Kafka/Pulsar for event streams, and RabbitMQ for business tasks:

@Service
public class UserEventPublisher {
    private final KafkaTemplate<String, UserEvent> kafkaTemplate;
    public UserEventPublisher(KafkaTemplate<String, UserEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }
    public CompletableFuture<Void> publish(UserEvent event) {
        String topic = "user-behavior-event";
        String key = event.userId();
        return kafkaTemplate.send(topic, key, event)
            .completable()
            .thenAccept(r -> log.info("sent {} {} {} {}",
                r.getRecordMetadata().topic(),
                r.getRecordMetadata().partition(),
                r.getRecordMetadata().offset(),
                event.eventId()))
            .exceptionally(ex -> { throw new EventPublishException("Kafka publish failed", ex); });
    }
}

Key design points:

Use acks=all and idempotent producer for high reliability.

Partition by userId to keep per‑user ordering.

Manual offset commit avoids “consume‑then‑fail” loss.

Suitable for throughput‑oriented pipelines, not fine‑grained business acknowledgements.

@Component
public class OrderTimeoutConsumer {
    private final OrderService orderService;
    public OrderTimeoutConsumer(OrderService orderService) { this.orderService = orderService; }
    @RabbitListener(queues = OrderMqConfig.ORDER_DEAD_QUEUE)
    public void onTimeout(OrderCreatedEvent event, Channel channel,
                          @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
        try {
            orderService.closeIfUnpaid(event.orderId(), event.eventId());
            channel.basicAck(tag, false);
        } catch (DuplicatedEventException ex) {
            channel.basicAck(tag, false);
        } catch (TemporaryDependencyException ex) {
            channel.basicNack(tag, false, true);
        } catch (Exception ex) {
            log.error("close order failed, orderId={}", event.orderId(), ex);
            channel.basicReject(tag, false);
        }
    }
}

RabbitMQ strengths highlighted: TTL + DLX for order timeout, explicit ACK/NACK handling, and full lifecycle control.

9. Engineering Enhancements

Partition design : choose ordering keys (orderId, userId, deviceId) to balance parallelism and ordering.

Idempotence : enforce unique business keys, DB unique indexes, Redis idempotency keys, or state‑machine checks.

Retry layering : instant retry in‑process, delayed retry via external queue, final dead‑letter handling.

Back‑pressure & rate limiting : configure consumer prefetch / max.poll.records, thread‑pool isolation, circuit‑breakers.

Schema governance : versioned message schemas, backward‑compatible additions, schema registry enforcement.

Observability : monitor production success rate, send latency (P95/P99), consumer lag, backlog depth, DLQ growth, broker resource usage.

10. Real‑World E‑Commerce Combination

Order creation – write to DB + outbox, CDC pushes OrderCreated to Kafka for downstream inventory, marketing, risk services.

Payment result – payment service publishes a command to RabbitMQ; fulfillment, SMS, and notification services consume independently.

User behaviour & telemetry – all events go to Kafka for real‑time analytics and data‑warehouse ingestion.

Merchant SaaS platform – multi‑tenant event bus built on Pulsar, providing isolated namespaces, quota enforcement, and tiered storage.

11. Capacity Planning & Load‑Testing Guidance

Key dimensions to size the system:

Average message size, peak QPS, sustained peak duration.

Acceptable latency, consumer concurrency, retention period.

Failure‑retry ratio and acceptable DLQ volume.

Example calculation: 200 k messages/s × 2 KB = 400 MB/s → 720 GB over a 30‑minute spike, clearly a Kafka/Pulsar scenario.

Load‑test focus:

Producer and consumer throughput limits.

Backlog recovery time after spikes.

Broker failure fail‑over and partition rebalance latency.

Consumer restart recovery speed.

System behaviour under retry storms.

12. Final Recommendation

Use Kafka when you need high‑throughput event streams, replay, and integration with Flink/Spark.

Choose RabbitMQ for business commands, fine‑grained ACK, DLQ, TTL, and complex routing.

Adopt Pulsar only if you require native multi‑tenant isolation, tiered storage, and have the ops capability to manage its components.

Most mature systems combine Kafka/Pulsar for data pipelines and RabbitMQ for task‑oriented workflows, linked by an Outbox + CDC consistency layer.