Mastering Redis Cluster: Scaling, Routing, and Fault Tolerance Explained

Significance of a Redis Cluster

Clustering solves four key problems: (1) higher QPS beyond a single‑master‑multiple‑slave setup, (2) data volume that exceeds a single machine’s capacity, (3) network traffic that would saturate a NIC, and (4) the need for buffering in offline computation.

CLUSTER MEET – Node Discovery

Basic Idea

Nodes start in an unknown, untrusted state. A node can be introduced to the cluster in two ways:

Administrator sends CLUSTER MEET <ip> <port> to force a node to meet another.

A known node shares its gossip list; if the receiver trusts the sender, it processes the gossip and initiates a handshake with the unknown node.

Because gossip is exchanged in periodic heartbeats, a single trusted‑node chain is enough for every node to eventually learn about all others.

Example

Assume a cluster with nodes A, B, C, D. Sending the following commands only to node A is sufficient:

CLUSTER MEET B-ip B-port
CLUSTER MEET C-ip C-port
CLUSTER MEET D-ip D-port

After A knows the others, its heartbeat gossip propagates the connections, forming a complete network within seconds.

Implementation Detail: MEET vs PING

When a node receives a MEET packet it treats the specified node as trusted and replies with a MEET packet (instead of a regular PING), forcing the receiving node to accept the sender as a trusted peer.

Slot Assignment

Redis Cluster has 16384 hash slots, evenly divided among master nodes. Each node reads/writes only the keys that map to its own slots.

All nodes exchange slot ranges so every node knows the full mapping.

Clients retrieve the slot map (via CLUSTER SLOTS) and route commands accordingly.

Cluster Scaling

Adding a New Node

Start a new Redis instance with cluster-enabled yes.

Handshake it with any existing node using CLUSTER MEET <ip> <port>.

Allocate slots to the newcomer (e.g., CLUSTER ADDSLOTS) and migrate the corresponding key‑value pairs. Data migration can be pipelined for efficiency.

Because the cluster uses a gossip protocol, once the new node meets a single member the whole cluster learns about it after a short period.

Removing a Node

Before shutting down a node, migrate its slots to other masters, then issue CLUSTER FORGET <node-id> so the remaining nodes drop the offline member.

Client Routing and Redirection

MOVED Redirection

If a client contacts a node that does not own the key’s slot, the node returns a MOVED error containing the target node’s address. The client must resend the command to the indicated node.

ASK Redirection

During slot migration the original node may still own the key temporarily. It returns an ASK error; the client must first send ASKING to the target node and then repeat the original command.

Smart Clients (e.g., JedisCluster)

On startup, execute CLUSTER SLOTS to build a local slot‑to‑node map and create a connection pool for each master.

Key → slot: slot = CRC16(key) % 16384. The client routes the command directly to the responsible node.

If a node becomes unreachable, the client receives a redirection, refreshes its slot cache, and retries (default up to 5 attempts, after which it throws “Too many cluster redirection!”).

Batch Operations on Multiple Keys

Commands that operate on multiple keys (e.g., MGET, MSET) must target keys that reside in the same slot.

Serial MGET

Iterate over each key, send a separate request to the responsible node, and aggregate results – simple but incurs O(keys) network latency.

Serial I/O with Grouping

Hash each key locally, group keys by node, then pipeline the grouped requests. This reduces round‑trips to O(nodes).

Parallel I/O

Spawn a thread per node and issue pipelined requests in parallel; overall latency equals the slowest node (O(max_slow(node))).

Hash Tags

Enclose a common substring in curly braces (e.g., {user}:id) so all such keys map to the same slot, allowing a single‑node MGET and eliminating cross‑slot errors.

Comparison

Serial MGET – simple, suitable for few keys, latency O(keys).

Serial I/O – simple, suitable for few nodes, latency O(nodes).

Parallel I/O – higher complexity, latency depends on the slowest node.

Hash tags – best performance, but requires careful key design and may cause data skew.

Fault Detection and Recovery

Failure Detection

Redis Cluster uses periodic PING / PONG messages (no Sentinel required) to exchange node status, slot ownership, and health.

Subjective down (PFAIL) : If a node does not receive a PONG within cluster-node-timeout, it marks the peer as PFAIL.

Objective down : When a majority of masters report a node as PFAIL, the cluster marks it objectively down and initiates failover.

Failover Process

Eligibility check : A slave is eligible if its link to the failed master is younger than cluster-node-timeout × cluster-slave-validity-factor (default 150 s).

Election : Among eligible slaves, the one with the greatest replication offset is preferred.

Promotion :

Execute SLAVEOF NO ONE to stop replication.

Remove the failed master’s slots with CLUSTER DELSLOTS and add them to the new master with CLUSTER ADDSLOTS.

Broadcast a PONG so the rest of the cluster learns the new topology.

Common Operational Issues

Cluster Integrity

cluster-require-full-coverage

defaults to yes ; all 16384 slots must be served for the cluster to be operational.

Setting it to no allows the cluster to serve partial data during node failures, but clients may see (error) CLUSTERDOWN The cluster is down.

Bandwidth Consumption

Gossip and heartbeat traffic generate ~2 KB slot array + ~1 KB status per 10 nodes per heartbeat.

Official guidance recommends keeping the node count below 1000 to avoid excessive bandwidth.

Pub/Sub Overhead

Publishing on any cluster node is broadcast to all nodes, which can significantly increase bandwidth.

Cluster Skew

Uneven slot distribution, large keys (bigkeys), or inconsistent memory settings cause data imbalance.

Tools: redis-trib.rb info <ip:port>, redis-trib.rb rebalance <ip:port> (use with caution), and redis-cli --bigkeys help detect and mitigate skew.

Read‑Write Separation

In cluster mode, replica nodes do not accept writes; reads can be redirected to the master owning the slot.

The READONLY command enables read‑only connections on replicas.

Pros and Cons of Redis Cluster

Limitations

Multi‑key operations (MGET, MSET, transactions, Lua scripts) require all keys to be in the same slot.

Only a single logical database (db0) is supported.

Replication is limited to one master‑slave level; no tree‑shaped replication.

When to Use

Redis Cluster provides horizontal scalability for large data volumes and high QPS workloads.

For many use cases, a single master with Sentinel or a simple master‑slave setup is sufficient and simpler.

References:

https://www.slideshare.net/iammutex/redis-cluster

https://zhuanlan.zhihu.com/p/105569485

https://sunweiguo.github.io/2019/02/01/redis/Redis-Cluster%E7%90%86%E8%AE%BA%E8%AF%A6%E8%A7%A3/

https://redis.io/topics/cluster-spec

http://trumandu.github.io/2016/05/09/RedisCluster%E6%9E%84%E5%BB%BA%E6%93%8D%E4%BD%9C%E6%8E%A2%E8%AE%A8/