Why Consistent Hashing Is the Key to Scalable Redis Clusters

When using Redis for high availability and performance, simple master‑slave replication or basic sharding can lead to inefficient data lookup, especially when data is randomly distributed across multiple cache servers.

Using Hash for Redis Sharding

Applying a straightforward hash function (e.g., hash(a.png) % 4 = 2) allows each key to be mapped to a specific server, eliminating the need to query all nodes.

Problems with Simple Modulo Hashing

If the number of cache servers changes—by adding or removing a node—every key’s location must be recomputed, causing massive cache invalidation and increased load on the backend database.

The Mystery of Consistent Hashing

Consistent hashing maps the entire 32‑bit hash space onto a virtual ring. Each server is hashed (using its IP or hostname) to a point on the ring, and a key is placed on the ring using the same hash function; the first server encountered clockwise is responsible for that key.

Fault Tolerance and Scalability

If a node fails, only the keys that map to the preceding node on the ring need to be redistributed, minimizing impact. Adding a new node similarly affects only the keys that fall between the new node and its predecessor.

Data Skew and Virtual Nodes

With few physical nodes, data can become unevenly distributed. Introducing multiple virtual nodes per physical server (e.g., Node A#1, Node A#2, …) spreads keys more uniformly across the ring.

Conclusion

Consistent hashing ensures that when cache nodes are added or removed, only a small subset of keys need to be relocated, providing high fault tolerance and easy scalability for distributed caching systems.