How to Choose the Right Number of Kafka Partitions for Optimal Throughput

Choosing the appropriate number of partitions for a Kafka topic is crucial for achieving optimal throughput while avoiding diminishing returns.

Increasing partitions can boost overall throughput, but beyond a certain threshold the throughput may drop. It is recommended to perform thorough throughput testing on the target hardware before production to identify the optimal partition count.

When a topic uses keys, Kafka assigns messages to partitions based on the key, ensuring that records with the same key land in the same partition, which is important for use cases such as log compaction and ordered consumption. Changing the number of partitions can break ordering guarantees for a given key, so it is best to decide the partition count at topic creation. For key‑heavy workloads, consider allocating more partitions to accommodate future growth, typically based on the projected two‑year throughput.

Some applications require strict ordering of all messages; in such cases setting the partition count to 1 leverages Kafka’s partition ordering to enforce topic‑wide order.

However, more partitions consume additional file descriptors, which are limited per process. Before increasing partitions, verify the current number of file descriptors used by the Kafka process.

Partition count also impacts system availability. Each partition has one leader and one or more replicas. If a broker fails, the leader partitions on that broker become temporarily unavailable while a new leader is elected. The larger the number of partitions, the longer the leader election and controller recovery may take, potentially extending the outage window.

For example, in a three‑broker cluster with 3,000 partitions (each with three replicas), a single broker failure makes 1,000 partitions unavailable. If each partition recovers in 5 ms, the total downtime is about 5 seconds. Adding more brokers can reduce the number of partitions per broker and mitigate this impact.

If the failed broker also hosts the controller, controller election is delayed until the new controller loads all metadata, which takes longer with more partitions, further slowing recovery.

More partitions also increase startup/shutdown time, log‑cleaning duration, and deletion time. Older client versions incur higher overhead with many partitions, though newer clients have mitigated this.

How to Choose the Right Number of Partitions?

Ultimately, selecting the partition count is a practical decision based on experience, Kafka’s characteristics, business requirements, hardware resources, and configuration. After setting the count, monitor and tune the topic as needed. A common rule of thumb is to set the number of partitions to a multiple of the broker count (e.g., 3, 6, 9 for a three‑broker cluster) based on estimated throughput. For very large clusters, additional factors such as hardware capacity and architectural considerations should be taken into account.