Understanding Kafka Partitions: Benefits, Trade‑offs, and Assignment Strategies

Kafka distributes a topic's messages across multiple partitions stored on different brokers, enabling high‑throughput producer and consumer processing; each partition acts as the smallest unit of parallelism.

Advantages of many partitions include parallel producer/consumer threads and higher aggregate throughput, but excessive partitions increase memory usage (producer batch buffers per partition), thread and socket overhead, file‑handle consumption, and longer leader election times, reducing high‑availability.

To choose a partition count, start with a single‑partition test, measure producer (Tp) and consumer (Tc) throughput, then compute partitions = T_target / max(Tp, Tc).

Message routing: by default Kafka hashes the message key (hash(key) % numPartitions) to select a partition; if the key is null, Kafka picks a partition randomly and caches the choice.

Consumer‑group semantics: each partition can be consumed by only one consumer thread within a group, though a thread may consume multiple partitions. Ideally, the number of consumer threads matches the number of partitions for maximum throughput.

Kafka provides two partition‑assignment strategies, selectable via partition.assignment.strategy:

Range : partitions are sorted and divided among consumer threads; early threads may receive more partitions, leading to imbalance.

RoundRobin : partitions from all subscribed topics are sorted by hash code and distributed cyclically across consumer threads, requiring equal num.streams per consumer and identical topic subscriptions.

Examples illustrate how a topic with 10 partitions is allocated among two consumers (C1, C2) under each strategy, highlighting the potential imbalance of Range and the more even distribution of RoundRobin.

Note: custom assignment strategies are not currently supported; users can only choose between range and round‑robin.