Spring Cloud + Kafka: 6 Common Pitfalls and How to Avoid Them

1. Silent Message Loss

Kafka’s default acks=1 can cause messages to disappear during a leader failover because the leader acknowledges before replication completes. The fix is to configure the producer with:

spring:
  kafka:
    producer:
      acks: all
      retries: 3
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5   // Kafka < 2.5, 10 for newer versions

These settings enforce acks=all, enable idempotence, and limit in‑flight requests, causing the application to fail fast rather than lose messages. On the consumer side, switch from automatic to manual offset commits to avoid duplicate processing.

2. Duplicate Processing

Kafka guarantees at‑least‑once delivery, so a message may be processed multiple times. A real case involved a points service that added points three times for a single order.

Solutions include:

Add a unique index on business tables, e.g., (order_id, type), to let the database reject duplicates.

Use Redis SETNX for fast de‑duplication.

When retries have side effects, store processing state in a dedicated message table with a state machine.

3. Message Order Chaos

Kafka only guarantees ordering within a single partition. An order‑status flow ( Created → Paid → Shipped) was split across three partitions, resulting in out‑of‑order consumption ( Shipped → Paid → Created) and downstream errors.

Fixes:

Send messages with a consistent key (e.g., orderId) so they hash to the same partition:

kafkaTemplate.send("order-events", orderId, message);

For order‑sensitive consumers, enforce single‑threaded processing ( concurrency=1) or implement an explicit queue.

4. Consumer Lag of 800 000 Messages

An alarm woke the team at midnight to discover 800 k messages lagging because no consumer‑lag monitoring existed.

Emergency actions:

Increase partition count and add more consumer instances to the same consumer group.

Temporarily enlarge the consumer thread pool.

Long‑term safeguards:

Monitor records-lag-max on the consumer side.

Watch broker metrics such as UnderReplicatedPartitions.

Track consumer thread‑pool health and pause/resume events.

Route permanently failing messages to a dead‑letter queue.

5. Serialization Breakdowns

The default KafkaTemplate serializer is StringSerializer. Using JsonSerializer with objects like LocalDateTime caused mismatched byte arrays after an upgrade.

A cross‑team incident arose when Java producers and Python consumers disagreed on date formats.

Recommended practices:

Agree on a global schema (Protobuf, Avro, or a fixed JSON contract).

Standardize time fields as epoch timestamps or ISO‑8601 strings.

Use a Schema Registry (Confluent, Apicurio, etc.) for multi‑language compatibility.

6. Treating Kafka Transactions as a Universal Remedy

Many assume Kafka’s transaction support can handle distributed operations like “order + inventory”. In reality, Kafka transactions only guarantee atomic writes within the Kafka cluster and the EOS three‑step (consume‑process‑produce) flow.

Any external side effect—database writes, RPC calls, Redis updates—lies outside Kafka’s transactional scope. For true distributed transactions, adopt dedicated solutions such as Seata or the Saga pattern.

Overall, Kafka’s default configuration is merely “usable”. Almost every production incident stems from an overlooked parameter. Documenting these pitfalls in team guidelines prevents repeated pain.