Why Kafka Still Delivers Out‑of‑Order Messages Even When Using the Same Key

Understanding Kafka Partition Ordering

Kafka guarantees only that messages are stored in order within a partition; the poll() call returns records sorted by offset, but this does not ensure that the consumer processes them in the same order.

Producer Retries Can Reverse Order

When max.in.flight.requests.per.connection is greater than 1, multiple requests may be in flight. If request 1 fails and is retried after request 2 succeeds, the broker may write request 2 first, resulting in offset B before A. This reorders messages such as payment before order creation.

请求 1 → 消息 A（用户下单）
请求 2 → 消息 B（用户支付）

Enabling the idempotent producer (Kafka 0.11+) adds a sequence number per producer‑partition pair, forcing the broker to place retried messages in the correct position. Configuration is simply:

# Producer configuration
enable.idempotence=true

Consumer Multithreading Breaks Order

Even though poll() returns records in offset order, dispatching them to a thread pool for parallel processing means that faster tasks finish earlier, producing a processing order like "payment → shipping → creation".

@KafkaListener(topics = "order_topic", groupId = "order-service")
public void onMessage(ConsumerRecord<String, String> record) {
    // Submit to executor for async handling
    executorService.submit(() -> orderService.process(record));
}

Setting concurrency = 1 forces a single consumer thread, preserving processing order at the cost of throughput.

Partition Expansion Changes Key‑to‑Partition Mapping

Kafka uses hash(key) % numPartitions to assign a partition. Expanding the number of partitions changes numPartitions, causing the same key to map to a different partition. Historical messages stay in the old partition, new messages go to the new one, breaking ordering across partitions.

int partition = Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

Therefore, once a partition count is chosen for a workload with ordering requirements, it should not be altered without migrating old data.

Practical Solutions

Enable the idempotent producer to keep order despite retries.

Use a single‑threaded consumer ( concurrency = 1) for strict ordering.

For mixed workloads, route messages with the same key to a fixed set of single‑thread executors, ensuring per‑key ordering while allowing parallelism across keys.

Plan the partition count based on peak consumer parallelism and avoid later expansion; if expansion is unavoidable, ensure all old partitions are fully consumed before adding new ones.

Final Takeaway

Kafka guarantees ordering only up to broker storage. Producer retry behavior, consumer concurrency, and partition scaling can all disrupt the end‑to‑end order, so developers must implement additional safeguards in their code.