7 Essential Kafka Design Patterns Every Engineer Should Master

1. Single‑Key Single‑Write Pattern

Applicable scenario: You need to preserve event order for each user, session, or entity.

Principle: Kafka guarantees ordering only within a partition. By routing all events with the same key to the same partition and using a single producer for that topic, order is maintained.

Example: A ride‑hailing app publishes driver location updates.

key = driver_id
value = { "lat": ..., "lng": ..., "timestamp": ... }

Tip: Use identifiers such as user_id, account_id, or driver_id as partition keys; avoid random partitioning when order matters.

2. Log‑Compacted Topic for Latest State

Applicable scenario: You need to keep the latest state of an entity.

Principle: A log‑compacted topic retains only the most recent message for each key, which is useful for rebuilding state (e.g., caches) from Kafka.

Example: A user‑profile service publishes updates.

key = "user-123"
value = { "email": "[email protected]" }

Even consumers that start later can reconstruct the latest profile information from the compacted log.

Tip: Suitable for user status, configuration, feature flags, etc.; combine with snapshots to accelerate recovery.

3. Multi‑Consumer‑Group Fan‑Out

Applicable scenario: Multiple downstream systems need to consume the same events.

Principle: Each consumer group in Kafka maintains its own offset, allowing a single producer to be read by many independent groups.

Example: A payment‑success event is consumed by an analytics team, an email‑notification service, and a fraud‑detection system, each using a different consumer‑group ID.

Tip: Do not overload a single consumer group with too many members, as they will compete for partitions; however, having many groups is safe—Kafka is built for it.

4. Retry and Dead‑Letter Topics

Applicable scenario: You need to handle transient or unrecoverable processing failures.

Principle: Instead of blocking or crashing, route failed messages to a retry topic (short delay, exponential back‑off) and, after a configurable number of attempts, to a dead‑letter topic (DLT) for permanent failures.

Example flow:

main‑topic --> processing‑service --> retry‑topic (on failure)
↓
dead‑letter‑topic (after 3 retries)

Tip: Include failure reasons in the DLT and actively monitor DLT size to prevent silent growth.

5. Exactly‑Once Processing (EOS) with Kafka Streams

Applicable scenario: You need true exactly‑once semantics—neither at‑least‑once nor at‑most‑once.

Principle: Kafka Streams achieves EOS by using an idempotent producer together with transactional writes to both Kafka and an external database.

Example: A banking service processes debit events and must avoid duplicate charges or credits.

builder.stream("debit-events")
       .transform(() -> new TransactionalProcessor())
       .to("ledger-entries");

Tip: EOS adds complexity; only enable it when truly required.

6. Schema Evolution with Avro + Confluent Schema Registry

Applicable scenario: Your messages evolve over time.

Principle: Avro combined with the Schema Registry lets you define schemas that can evolve safely while maintaining backward/forward compatibility.

Example schema:

record User {
  string user_id;
  string email;
  string phone; // new optional field
}

Both producers and consumers validate data against the schema before sending or receiving.

Tip: Always enforce compatibility rules (default: backward) and keep clear version documentation.

7. Choreography vs. Orchestration

Applicable scenario: You have a multi‑service workflow (e.g., order processing).

Principle:

Choreography: Services listen for events and react independently (loose coupling).

Orchestration: A central orchestrator commands the flow, providing explicit control.

Choreography example: Order service emits OrderPlaced, inventory service listens and emits InventoryReserved, payment service listens and emits PaymentCompleted.

Orchestration example: An orchestrator invokes inventory and payment services sequentially and waits for their results.

Tip: Choreography scales better but is harder to debug; orchestration is easier to debug but introduces tighter coupling.

Common Pitfalls

Too many partitions: Increases startup time and rebalancing overhead.

Large messages (>1 MB): Can overwhelm downstream systems; consider object storage for big payloads.

Messages without a key: Lose ordering guarantees; always include a key for critical entities.

Long consumer processing time: Blocks rebalancing; keep processing fast and stateless when possible.

Final Thoughts

Kafka is merely a pipeline—how you design its usage determines reliability, observability, and maintainability. Senior engineers should focus on safe retry flows, sensible schema design, and avoiding chaotic event‑driven architectures. Remember: a design that handles 1 K TPS may crumble at 100 K TPS.