Replication and Partitioning Mechanisms in Redis, Kafka, and Elasticsearch

Based on the book Designing Data‑Intensive Applications (DDIA), this article explores the replication and partitioning mechanisms of common components such as Redis, Kafka, and Elasticsearch to deepen understanding of distributed‑system fundamentals.

DDIA Replication and Partitioning : Replication consists of a leader (master/primary) and followers (slave/replica). It can be single‑leader, multi‑leader, or leaderless, and uses synchronous, asynchronous, or semi‑synchronous confirmation, with replication lag being the main challenge.

Partitioning : Partition (or shard) splits large datasets into smaller pieces. Key‑value partitioning methods include range partitioning and hash‑based partitioning. Secondary indexing can be local (per partition) or global, and issues such as load skew, hotspots, and rebalancing may arise.

Redis Analysis : In cluster mode Redis defines 2^14 = 16384 slots; a key’s slot is determined by CRC16(key) % 16384 , making the slot count immutable. Slot rebalancing must be performed manually or with third‑party tools. Replication follows a master‑slave model (asynchronous by default) and can be tuned with parameters like min-replicas-to-write , min-replicas-max-lag , and replica-read-only .

Kafka Analysis : Kafka nodes are called broker and partitions are called partition . Partition count can only increase. Assignment strategies include round‑robin, hash‑based, and custom. Replication is per‑partition with ISR (In‑Sync Replicas). Producer durability is controlled by the acks setting (0, 1, all) and min.insync.replicas . Follower partitions do not serve consumer reads.

Elasticsearch Analysis : Documents are stored in shards, each having a primary and replica shard. Shard allocation uses a hash of the document ID modulo the number of primary shards, which cannot be changed after creation. Replication is asynchronous and can be awaited with wait_for_active_shards . Each replica can serve reads. Elasticsearch builds a local inverted index per shard; search types (e.g., query_then_fetch vs dfs_query_then_fetch ) affect scoring accuracy and performance. Routing parameters can improve result consistency.

Summary : By analyzing Redis, Kafka, and Elasticsearch, the article demonstrates how partitioning (often hash‑based) and replication (typically single‑leader asynchronous) are designed and implemented in distributed systems, highlighting trade‑offs between performance, complexity, and consistency.