Redis Cluster Deep Dive: Sharding, Slot Migration, and Failover Explained
This article provides a comprehensive technical analysis of Redis Cluster, covering its fixed 16384-slot partitioning, slot migration mechanics, MOVED vs ASK redirections, Gossip-based failure detection, replica election, hash tags, cross-slot limitations, and operational best practices with real-world incident case studies.
Introduction
Redis Cluster solves the problem of a single instance's memory ceiling (tens of GB) and write throughput ceiling (hundreds of thousands QPS) by splitting data across multiple machines. Instead of consistent hashing, Redis uses a fixed 16384-slot partition table. Each node claims a subset of slots, and clients holding the full table can compute the target node locally. This design introduces unique mechanisms: slot migration, MOVED vs ASK redirections, Gossip heartbeats, and replica election.
Three key facts to remember:
Slots are the minimum unit of routing, not storage — data still exists per key on a node; slots only map "which node owns this key".
The cluster has no central node; every node builds a global view via Gossip, which may be temporarily inconsistent — this is why MOVED and ASK exist.
Failover is initiated by replicas themselves through election, not by an external coordinator.
Understanding these three points explains all "strange behaviors" of Redis Cluster.
Three Incidents: Three Ways to Die
Incident 1: Scaling Out, P99 Latency Spikes from 5ms to 800ms
Before a major promotion, a 3-master cluster was expanded to 6 masters using redis-cli --cluster reshard. The tool uses pipelined batch MIGRATE (default pipeline 10, hundreds of keys per batch). However, MIGRATE blocks on the main thread : it serializes the key, connects to the target, sends RESTORE, waits for reply, then deletes locally. A 10KB key takes ~0.3ms round-trip; 10,000 keys accumulate 3 seconds of pauses, fragmented into normal requests. During migration, client P99 latency increased 160x while main-thread CPU was only 30% — the bottleneck was network wait, not compute. Scaling operations must be scheduled in maintenance windows; they are not run-anytime scripts.
Incident 2: One Master Down, Entire Cluster Unavailable
A 3-master/3-replica cluster lost a master host. The replica should have been promoted, but it had restarted the day before and was still in full sync, so its replication offset lagged too far. It was deemed "data too stale" (failed cluster-replica-validity-factor) and abstained from election. Consequently, 5461 slots lost ownership. Worse, the cluster was configured with cluster-require-full-coverage yes — any uncovered slot causes the whole cluster to reject all requests with CLUSTERDOWN. Only 1/3 of machines failed, but 100% of business became unavailable ; the other two healthy masters served zero requests.
Incident 3: Cross-Slot Operations — Half Succeed, Half Error
An order service used MGET order:1001 order:1002. In test (only three keys, all in same slot) it worked. In production, users fetching ten orders triggered CROSSSLOT as soon as two keys hashed to different slots. Even more subtle: MULTI and Lua scripts spanning slots return errors buried in EXEC replies; unchecked code assumes success. In cluster mode, key affinity must be explicitly expressed via hash tags, not left to chance.
Technical Analysis: Why 16384 Slots?
2.1 Three Problems with Consistent Hashing
Classic consistent hashing maps nodes and keys to a ring; a key finds the first node clockwise. Advantages: adding/removing nodes only affects adjacent ranges. But three engineering problems exist:
Data skew — with few nodes, distribution is uneven; virtual nodes (hundreds per physical node) are needed, making the mapping table itself metadata that must be synchronized.
Clients cannot compute locally — to know "which node owns this key", the client must know all node positions on the ring. This information changes and must be synced; during sync, computed answers are wrong.
Cannot precisely express "move this data segment" — migration can only estimate by hash range; boundaries are fuzzy.
The slot partition table solves all three at once: Fixed 16384 slots, naturally uniform (each slot ~1/16384 of data). Clients with the "which slots belong to which node" table can compute the target locally with one modulo operation — no need to ask anyone. Migration granularity is exact to the slot, with clear semantics before and after. The trade-off: the partition table itself must be synchronized — hence Gossip and MOVED exist to propagate changes and correct stale views.
2.2 Origin of the Number 16384
Not arbitrary. Author antirez gave three reasons on the mailing list:
Heartbeat packets must fit the slot bitmap — each node includes a 16384-bit bitmap (2KB) in PING/PONG. With many nodes, packet size multiplies; 65536 slots would be 8KB, exceeding Gossip bandwidth budget.
Practical cluster size limit — Redis Cluster design max nodes ~1000. 16384 slots / 1000 nodes = ~16 slots per node, sufficient granularity.
Bitwise operation efficiency — 16384 = 2^14, so modulo can be done with & 0x3FFF instead of division; compressed bitmap stays small at typical scales.
Conversely, this is the source of the cluster scale limit. Not "Redis doesn't support more nodes", but "too few slots = not enough to divide; too many slots = heartbeat too heavy". Understanding this trade-off explains why 200-node clusters are rare — not impossible, but Gossip bandwidth and convergence latency outweigh benefits. The first constraint on cluster scale is never memory; it's the heartbeat.
2.3 Key to Slot: CRC16 and Hash Tags
Algorithm in one line: CRC16(key) & 0x3FFF (low 14 bits). The real engineering lever is hash tags — if a key contains curly braces, only the content of the first pair is hashed. This makes "related keys land in the same slot" programmable:
user:{1001}:name -> h("1001")
user:{1001}:cart -> h("1001")
order:{1001}:list -> h("1001")
/* Curly brace rules */
{} empty tag -> whole key hashed
no { -> whole key hashed
a{b}c{d} exists -> only "b" used
{a{b}} -> only "a{b" used
name:} -> whole key (no {)Hash tags are the #1 engineering practice in cluster mode and the most common pitfall. Correct usage: tag by "transaction boundary" — all attributes, queues, locks, counters of an order must share a slot, otherwise a MULTI or Lua script fails. Misuse: "global same tag" — all keys forced into one slot, degenerating the cluster to a single instance plus Gossip overhead. Judgment criterion: tag cardinality should equal your business primary key cardinality, not a constant.
2.4 Cluster Bus: Two Ports per Node
Each node listens on two ports:
Client port 6379 (RESP protocol)
Cluster bus port 16379 (port + 10000, binary protocol)
Channels are fully isolated: client port handles business commands; bus port handles inter-node Gossip. Isolation prevents node-to-node probing from blocking client requests and vice versa, and allows separate network policies. Many production incidents stem from firewalls opening only 6379 but not 16379, causing nodes to never handshake.
Gossip rhythm: Each second, randomly pick 5 known nodes (prioritizing longest silent) and send PING . Each PING carries a gossip section with ~1/10 of nodes' status (IP, port, slot bitmap, flags). Receivers reply PONG and record the info. Views spread via "neighbor's neighbor", achieving convergence time logarithmic in node count , not O(N) broadcast.
2.5 MOVED vs ASK: Two Words, Two Semantics
Favorite interview question, often answered wrong.
MOVED means "this slot permanently moved; go ask that node, and never ask me again" — client should update local slot table to point the slot to the new node.
ASK means "this slot is migrating out, but this key hasn't moved yet / already moved; try this command on the target node, but don't update your cache" — because migration isn't finished, slot ownership hasn't changed; updating the table would be wrong.
ASK's companion is the ASKING command. On receiving ASK, client must first send ASKING to the target, then the original command. The target marks the connection with a one-time flag allowing access to a slot not yet officially owned; without this flag, the target sees the slot isn't its own and returns MOVED, creating an infinite loop. The one-time design is critical: the flag clears after one command, preventing temporary authorization from becoming permanent privilege escalation.
2.6 Consistency Boundaries: What Cluster Does NOT Provide
Cold water first: Cluster solves capacity and throughput, not consistency. Three things it explicitly cannot give:
Strong consistent reads — master-write replica-read remains async replication; replicas may serve stale values; failover may lose the last writes.
Cross-slot atomicity — MULTI and Lua are restricted to a single slot; cross-slot means no atomicity.
Cross-key range queries — SCAN only iterates one node's keyspace; "scan all" requires merging N cursors manually, with no point-in-time consistent snapshot.
Treating these as "defects" is wrong; they are the inevitable cost of sharding. Once data is split across machines, a "global view at a single instant" ceases to exist — you either pay for distributed transactions (Redis doesn't) or accept eventual consistency at the application layer. Architecture decisions should follow: first ask "can my data be sharded, and do I still need cross-shard transactions?", then ask "should I use Cluster?". Reverse the order, and you'll pile patches like "read twice and compare" in business code.
Core Workflows: One GET and One Migration
3.1 A GET Request's Full Journey
Assume client starts with empty slot table. GET user:{1001}:name proceeds:
1 Client connects to any seed node
2 Local slot calc: CRC16("1001") & 0x3FFF
3 Suppose result 5474
4 Slot table empty, send CLUSTER SLOTS
5 Receive 16384-slot ownership, build local table
6 Find 5474 belongs to node B, connect B
7 B executes, returns result
8 (If B replies -MOVED 5474 C)
9 Update local table: 5474 -> C
10 Resend command to C, success
11 (If B replies -ASK 5474 C)
12 Do NOT update table, send ASKING to C
13 Resend original command, successStep 4 is the most overlooked performance point. Clients must cache the entire table and only update one slot on MOVED , not query CLUSTER SLOTS per request. Mature clients (JedisCluster, Lettuce) do exactly this, with periodic full-table refresh as fallback. The most common mistake in custom clients: "ask for slot table on every command" — effectively adding an extra round-trip per request, halving throughput.
3.2 Migrating One Slot: Seven Steps
Many think "migrate slot" is one command; actually it's a 7-step protocol where source and target each change state, move keys one batch at a time, then both finalize . Understanding these steps explains why ASK appears during migration:
① Target: SETSLOT 5474 IMPORTING <source-id>
② Source: SETSLOT 5474 MIGRATING <target-id>
③ Source: CLUSTER GETKEYSINSLOT 5474 100
④ Source: MIGRATE targetIP 6379 "" 0 5000 KEYS k1 k2 ...
⑤ Repeat ③④ until slot has no keys
⑥ Target: SETSLOT 5474 NODE <self-id>
⑦ Source: SETSLOT 5474 NODE <target-id>
Then broadcast; whole cluster updates slot tableSteps ③–⑤ are the ASK window. During this period, source still considers the slot its own (hasn't done SETSLOT NODE), but target is already IMPORTING. Client accesses:
Key still on source → source serves normally.
Key already migrated → source doesn't find it, but knows "this slot I'm migrating out", so returns ASK pointing to target. Target, being in IMPORTING, accepts the command with the ASKING flag.
The slot ownership never changed during this window, so MOVED cannot be returned — this is the entire reason ASK exists.
3.3 Failover: From PFAIL to New Master Takeover
Cluster failure detection has two levels, similar to Sentinel's SDOWN/ODOWN but implemented differently:
PFAIL = single node's private opinion (no PONG within cluster-node-timeout).
FAIL = cluster consensus — a node spreads PFAIL via Gossip; when more than half of masters report the same node as PFAIL, some node marks it FAIL and broadcasts, triggering replica election.
T+0 Master A stops responding
T+15s Nodes mark PFAIL (node-timeout)
T+17s Majority of masters confirm, mark FAIL
T+17s Broadcast FAIL, replicas start election
T+17s A's replicas calculate delay D:
500ms + rand(0-500) + rank * 1000ms
T+18s Earliest replica requests votes
T+18s Needs majority of masters' votes
T+18s Elected, epoch +1
T+18s Takes over all of A's slots
T+18s Broadcasts PONG, cluster updates tablesThe rank in the delay formula is the cleverest part. Each replica sorts by replication offset difference from the failed master; most up-to-date replica gets rank 0, each step behind adds 1 second . This ensures "the replica with the freshest data initiates election first" while avoiding vote splitting from simultaneous campaigns. Random 0-500ms prevents same-rank contention; 500ms base allows FAIL message propagation time. Together they guarantee "fastest + freshest + no conflict".
3.4 Network Partition: Minority Side Stops Itself
Master-replica uses min-replicas-to-write to limit damage; Cluster has a harder rule: If a node loses contact with "majority of masters" for several multiples of cluster-node-timeout , it proactively stops accepting writes and returns CLUSTERDOWN . This is the cluster's "minority self-stop" — it assumes the majority side will elect a new master and continue; the minority must yield, otherwise split-brain occurs.
This rule explains a hard constraint often violated in deployments: master count must be odd, spread across at least three failure domains. With 3 masters, partition 2 vs 1: the side with 2 has majority and can elect; the side with 1 self-stops. With 2 masters, partition 1 vs 1: neither achieves majority, both stop — cluster totally unavailable, worse than standalone. Same math as Sentinel's "at least three", but Cluster bakes it into node self-protection logic.
Core Source Code: Five Key Logic Sections
4.1 keyHashSlot: Two-Pass Scan for Curly Braces
/* cluster.c simplified */
unsigned int keyHashSlot(char *key, int keylen) {
int s, e;
for (s = 0; s < keylen; s++)
if (key[s] == '{') break;
if (s == keylen)
/* no { */
return crc16(key,keylen) & 0x3FFF;
for (e = s+1; e < keylen; e++)
if (key[e] == '}') break;
if (e == keylen || e == s+1)
return crc16(key,keylen) & 0x3FFF;
return crc16(key+s+1, e-s-1) & 0x3FFF;
}Three returns cover three edge cases. No curly braces, empty braces ( {}), or only left brace without right — all fall back to hashing the whole key. Only when { has content and a matching } is found does the tag apply. Note the second scan starts at s+1 and stops at first } — no nesting; {a{b}} yields tag a{b. This "simple to the point of crude" rule is exactly why all clients can replicate it accurately.
4.2 getNodeByQuery: Return ASK or MOVED
/* Decision when processing a command */
slot = keyHashSlot(key, len);
n = server.cluster->slots[slot];
/* Case 1: slot migrating out, key no longer on this node */
if (n == myself &&
migrating_slots_to[slot] &&
key not exist on this node)
return CLUSTER_REDIR_ASK;
/* Case 2: slot not owned by this node */
if (n != myself)
return CLUSTER_REDIR_MOVED;
/* Case 3: this node is IMPORTING, client sent ASKING flag */
if (importing_slots_from[slot] &&
(c->flags & CLIENT_ASKING))
execute normally, no redirectThe order of these three branches is the complete routing logic. Note case 1's condition "key not exist on this node" — if key is still present, source serves directly, no redirect; that's why most requests during migration remain normal. Only when the key has already been moved does ASK redirect to the target. Case 3 implements the temporary grant: IMPORTING state + ASKING flag, both required; missing either returns MOVED.
4.3 Gossip: Convergence Art of 5 PINGs per Second
Each node in clusterCron does four things per second: pick 5 least-recently-contacted nodes to PING, check if PFAIL needed, check if election needed, check if slot config update needed . PING carries three payloads: sender's slot bitmap, a gossip section (random ~1/10 of nodes' status), and current config epoch. Gossip section uses random sampling instead of full broadcast — the key to the design — keeping per-message size constant, at the cost of needing several rounds to converge.
Convergence speed empirical: ~10 nodes, slot change propagates in 1-2 seconds; hundreds of nodes, several seconds. During this window, nodes have inconsistent views; clients may receive stale MOVED — but MOVED keeps correcting, eventual consistency; business sees "occasional extra redirect", not errors. Real danger: "network partition causes two partitions each thinking the other is down" — brief split-brain, same root cause as master-replica brain split.
4.4 Replica Election: Rank Decides Who Acts First
/* clusterHandleSlaveFailover() simplified */
delay = 500 + rand() % 501;
/* rank: sort by offset diff, 0 = freshest */
rank = 0;
for (each sibling replica o)
if (o->offset > my_offset)
rank++;
delay += rank * 1000;
/* After delay, request votes */
if (mstime() < failover_auth_time)
return;
clusterRequestFailoverAuth();
/* Majority of masters must agree */
if (votes >= (master_count/2 + 1))
take over slots, epoch +1Here rank is not "ranking" but "count of replicas with newer data than me". Rank 0 = I'm freshest, jump at 500ms; rank 2 = two replicas are fresher, wait 2.5s. The brilliance: no coordination needed — each replica computes independently, yet the freshest naturally goes first. It also plugs a hole: a lagging replica winning would lose data; rank makes it lose the time race to fresher siblings.
4.5 Migration State: Two Arrays Remember "What I'm Moving"
Each node maintains two pointer arrays of length 16384: migrating_slots_to — "this slot I'm moving to whom" importing_slots_from — "this slot I'm receiving from whom"
These two arrays are the physical carriers of ASK semantics — without them, a node cannot distinguish "slot not mine" (return MOVED) from "slot I'm moving" (return ASK). Both states must be cleared by both parties after migration, and order matters: target first SETSLOT NODE self (officially takes over), then source SETSLOT NODE target (officially releases). Reverse order creates a "both think it's not theirs" gap; clients get MOVED pointing to a node that denies ownership.
Advanced Topics: Seven Accounts to Calculate Upfront
5.1 Correct Rhythm for Slot Migration
Iron law: never migrate during business peak. MIGRATE is synchronous blocking on the source main thread: serialize → connect target → RESTORE → wait reply → DEL local. Single key latency ≈ one network RTT.
Control batch key count and byte size — --cluster-pipeline default 10, can reduce to 1-5 for smoother latency curve.
Total migration time ∝ (key count × average RTT), not data volume — many small keys are slower.
Large keys must be handled separately — a 100MB hash serializes in one go, causing second-level block; must split large keys before migration .
Confirm replicas are caught up before migrating — otherwise after migration, if master fails, new master has incomplete data.
5.2 Should cluster-require-full-coverage Be Off?
Default yes: any uncovered slot → whole cluster refuses service. Stance: "rather fully unavailable than partially available", because partial availability lets business read incomplete data unknowingly. Setting no makes missing-slot requests return CLUSTERDOWN while other slots serve normally — Incident 2 degrades from "full outage" to "1/3 data failing" . Decision key: Can your business tolerate "some keys unavailable while others work"? Yes (most caching) → turn off, maximize availability. No (financial consistency) → keep yes, make failure explicit.
5.3 Four Solutions for Cross-Slot Operations
Hash tag convergence — canonical; tag related keys to same slot; MGET, MULTI, Lua all work; cost: those keys must share slot (cannot spread across nodes).
Client-side split — split MGET by slot, fire parallel single-key requests, merge results; sacrifice one RTT for parallelism; suitable when keys need no atomicity.
Pipeline instead of batch commands — pipeline can contain keys from different slots; client groups by node and sends; semantically "multiple independent commands", no atomicity.
Redis 7 Functions — still single-node execution; cross-slot problem unchanged.
Beware "pseudo-solutions that appear to work": adding a proxy (Twemproxy-style) to make cluster look like standalone, proxy aggregates cross-node — introduces new single point and latency, and transactions still can't be atomic across nodes.
5.4 Three Client Types and Selection
Smart Client — caches slot table locally, connects directly to target, updates on MOVED. JedisCluster, Lettuce cluster mode. Pros: zero extra hops, lowest latency. Cons: each client holds a view; connection count explodes with many nodes (N clients × M nodes).
Proxy — clients connect to proxy, proxy forwards. Pros: simple clients, controllable connections. Cons: proxy is bottleneck and single point; extra hop latency.
Hybrid — proxy as unified entry, smart routing inside — common where proxy infrastructure exists.
Selection criterion: (app instances × node count) > 5000 connections? If yes, consider proxy or connection pooling.
5.5 Cluster Scale and Heartbeat Cost
Each node sends 5 PINGs/sec, each ~2KB slot bitmap + gossip section. Single node heartbeat bandwidth ~10-20KB/s ; 100 nodes = 2MB/s internal traffic — seems small, but it's constant background traffic growing linearly with node count . Worse: failure detection convergence time grows with node count — more nodes means more Gossip rounds for "majority of masters confirm PFAIL", slowing failover. Hence production clusters usually stay under a few dozen masters; beyond that, "multiple clusters + application-level sharding" is preferred over one giant cluster.
5.6 Six High-Frequency Operational Pitfalls
FLUSHALL only clears one node — must run on every node; scripts connecting only to seed node cause "half cleared" accidents.
CLUSTER FORGET has 60-second cooldown — forgotten node ID enters blacklist; retries within 60s error; many think command failed and repeat.
CLUSTER RESET has soft/hard — hard regenerates node ID and clears slots = "this machine becomes a new node"; misuse causes slot loss.
Replicas don't auto-rebalance — cluster-replica-no-failover and cluster-migration-barrier control whether orphaned replicas migrate to master-less masters; default is no, requires manual intervention.
Node ID binds to IP — in containers/NAT, node announces internal IP; clients can't connect; must set cluster-announce-ip.
Slot table vs actual ownership mismatch — manual SETSLOT without broadcast leaves nodes with divergent tables; symptom: "random MOVED to wrong node"; fix only via CLUSTER NODES full audit.
5.7 When NOT to Use Cluster
Three clear cases:
Data fits in one large-memory machine — single instance 64GB covers it; Cluster just triples ops complexity with zero gain.
Heavy reliance on multi-key atomic ops — lots of MULTI, cross-key Lua, range queries; migrating to Cluster triggers massive rewrite; single-master multi-replica is cheaper.
Team lacks dedicated Redis ops expertise — Cluster troubleshooting requires reading CLUSTER NODES column by column, distinguishing PFAIL/FAIL, judging slot coverage; misoperations ( RESET / FORGET / SETSLOT) can directly cause data unavailability.
Conversely, clear signals to adopt Cluster: single instance memory near limit, write QPS near single-thread ceiling, or need horizontal scaling without downtime. Any one of these makes Cluster's complexity worth paying. The yardstick: "Is scaling demand already a norm?" — once a year, manual migration suffices; once a month, Cluster must be treated as infrastructure. This is not technology selection; it's cost selection.
Learning Path
Layer 1 (Can build & run): Spin up 6 nodes (3 master/3 replica) locally, run --cluster create, use CLUSTER NODES to understand every column (id/ip:port/flags/master/ping-pong/epoch/state/slot).
Layer 2 (Can calculate & migrate): Manually migrate one slot, walk the 7-step protocol; use CLUSTER KEYSLOT to find a key in migration, deliberately GET → MIGRATE → GET, observe ASK and ASKING effects.
Layer 3 (Can judge & rescue): Kill a master, watch CLUSTER NODES for PFAIL→FAIL→election→takeover; toggle cluster-require-full-coverage both ways, compare business impact.
Layer 4 (Can modify): Read cluster.c 's clusterProcessPacket and getNodeByQuery, map MOVED/ASK decision branches to slot migration state tables.
Interview Questions (with Answers)
Q1 · Why 16384 slots instead of consistent hashing?
Consistent hashing requires clients to know all node positions on the ring to compute ownership; metadata sync is hard, migration boundaries fuzzy, skew prone. Slot table is fixed 16384 mapping; clients with table compute locally with one modulo, no need to ask anyone; migration granularity exact to slot, clear semantics. 16384 comes from three constraints: heartbeat packet must fit 2KB slot bitmap (65536 = 8KB), design limit ~1000 nodes (16 slots each enough), 2^14 allows bitwise modulo.
Q2 · How is slot calculated? What is hash tag for?
CRC16(key) & 0x3FFF. If key contains curly braces with non-empty content, only first brace content is hashed — that's hash tag. Purpose: force related keys into same slot, enabling MGET, MULTI, Lua in cluster. Edge cases: {} treated as empty, missing } invalid, only first {} used, no nesting ( {a{b}} tag is a{b).
Q3 · Difference between MOVED and ASK?
MOVED = slot ownership permanently changed; client should update local table and retry. ASK = slot migrating, ownership unchanged, just this key temporarily not here; client sends ASKING then retries, does NOT update table. Decision based on node's migrating_slots_to / importing_slots_from tables: key already moved → ASK; slot fully transferred → MOVED.
Q4 · Why send ASKING after receiving ASK?
Target node is only IMPORTING; slot nominally not its own yet. Direct access would return MOVED again, creating a loop. ASKING sets a one-time flag on the connection (clears after one command), allowing access to an importing slot. One-time is crucial to prevent temporary grant becoming permanent privilege.
Q5 · How many ports? Why two?
Client port 6379 and cluster bus port 16379 (port + 10000). Former runs RESP business commands; latter runs binary Gossip (PING/PONG/MEET/FAIL). Isolation lets node probing and client requests not block each other; network policies separable. Most common production issue: firewall allows only 6379, nodes never handshake.
Q6 · How does Gossip propagate? Why doesn't it saturate bandwidth?
Each node picks 5 least-recently-contacted nodes per second, sends PING with 2KB slot bitmap and a gossip section (random ~1/10 nodes' status). Sampling instead of full broadcast keeps message size constant; convergence rounds logarithmic in node count. Trade-off: convergence latency — ~10 nodes 1-2s, hundreds of nodes several seconds.
Q7 · PFAIL vs FAIL?
PFAIL = single node's private view: no PONG within cluster-node-timeout, just records, no action (like Sentinel SDOWN). FAIL = consensus: a node spreads PFAIL via Gossip; when >half of masters report same node PFAIL, it's marked FAIL and broadcast, triggering replica election. Prevents "my network broken, so I trigger failover".
Q8 · How is replica election delay calculated? What is rank?
Delay = 500ms + random 0-500ms + rank × 1000ms. Rank = "number of sibling replicas with larger offset than me" (i.e., fresher data = smaller rank). Ensures freshest replica campaigns first; random avoids same-rank contention; 500ms base gives FAIL propagation time. No coordination needed — each replica computes independently, yet freshest naturally goes first.
Q9 · How many votes does a replica need to win?
Majority of masters (master_count/2 + 1) votes, plus own vote, and epoch must increment. Only masters vote; replicas don't. Majority ensures no two new masters elected simultaneously; config epoch invalidates old-term votes.
Q10 · cluster-require-full-coverage yes or no?
yes= any slot uncovered → whole cluster refuses; stance "rather fully down than partially up" to avoid silent incomplete reads. no = only missing slots fail, others serve. Caching workloads usually set no for max availability (1/3 data failing better than full outage); financial workloads keep yes to surface failures.
Q11 · MGET cross-slot returns CROSSSLOT, how to solve?
Four solutions: 1) Hash tag to co-locate keys; 2) Client splits by slot, parallel requests, merge; 3) Pipeline instead of batch, client groups by node, no atomicity; 4) Proxy aggregation (adds hop and single point). Choice depends on "do these keys need atomicity?" — yes → hash tag; no → split or pipeline.
Q12 · Transaction and Lua limitations in cluster?
All keys in MULTI / EXEC and EVAL must be in same slot, else error. Because transactions and scripts execute atomically on a single node; cross-node atomicity impossible. Practice: tag keys of same business entity with hash tag (e.g., user:{1001}:*) to align atomicity boundary with slot boundary. Note: MULTI cross-slot error may only appear in EXEC reply; business code must check.
Q13 · Complete steps to migrate a slot?
1) Target SETSLOT IMPORTING source-id; 2) Source SETSLOT MIGRATING target-id; 3) Source GETKEYSINSLOT fetch batch; 4) MIGRATE move batch; 5) Loop until slot empty; 6) Target SETSLOT NODE self; 7) Source SETSLOT NODE target, then broadcast. Order cannot reverse: target takes over first, source releases after; otherwise "both deny ownership" gap appears.
Q14 · Why does MIGRATE cause latency spikes?
MIGRATEruns synchronously on source main thread: serialize key → connect target → send RESTORE → wait reply → DEL local. Single key latency ≈ one network RTT. Batched migrations slice these RTTs into countless fragments interleaved with normal requests; cumulative pause observable but CPU low (bottleneck is network wait). Hence migration must be windowed, pipeline tuned, large keys split first.
Q15 · What happens to client requests during migration?
Three cases: key still on source → source serves normally; key already migrated → source returns ASK to target, client sends ASKING then executes on target; migration complete and slot officially transferred → source returns MOVED, client updates local table. No errors, just extra redirect; migration impact is "latency jitter", not "errors".
Q16 · Why reshard after scaling? Why not auto-balance?
New nodes join with 0 slots; must be explicitly assigned. Existing nodes won't auto-release slots because migration is a costly blocking operation; auto-balance would cause incidents during peak. Redis provides redis-cli --cluster rebalance for semi-auto (with --threshold and --pipeline controls), but trigger timing remains human decision. Classic "dangerous ops left to humans".
Q17 · Do cluster nodes replicate data to each other?
No. Cluster only does "sharding", not "mutual replication". Replica relationship remains master-replica replication channels, orthogonal to Cluster topology. So "3 master 3 replica" = 6 nodes, 3 replication pairs, not 6 nodes backing each other. Understanding this clarifies capacity: cluster usable capacity is half (if 1 replica per master).
Q18 · Can SELECT and pipeline be used in cluster mode?
SELECTonly works for DB 0; multi-DB concept removed (sharding replaces DBs). Pipeline works but must target same node — client must group by slot first, otherwise reply parsing misaligns. Mature clients auto-group.
Q19 · Is there a cluster size limit? What constrains it?
Soft limit ~1000 nodes; production usually stays at few dozen masters. Three constraints: 1) 16384 slots must divide evenly; too many nodes → few slots each, migration granularity loses meaning. 2) Heartbeat bandwidth grows linearly with nodes (10-20KB/s per node). 3) Failure detection convergence grows with nodes; "majority confirm" needs more Gossip rounds. Beyond that, prefer multiple clusters + app-level sharding.
Q20 · Your default production Redis Cluster setup?
3 master / 3 replica minimum (across 3 AZs), cluster-node-timeout 15s, cluster-require-full-coverage no for caching, explicit cluster-announce-ip, both 6379 and 16379 open; business keys uniformly hash-tagged to entity ID; Smart Client with slot cache and MOVED retry; scaling in low-traffic windows, small pipeline, large keys pre-split; monitor CLUSTER INFO fields: cluster_state, cluster_slots_assigned, cluster_known_nodes, and per-node PFAIL counts.
Cheat Sheet: 20 Questions in One Line
16384 slots, CRC16 & 0x3FFF<br/>{tag} forces related keys same slot<br/>Two ports: 6379 + 16379<br/>MOVED updates cache, ASK updates once<br/>ASK requires ASKING (one-time)<br/>Migration 7 steps: take over first, release after<br/>PFAIL private, majority makes FAIL<br/>Delay = 500 + rand + rank*1000<br/>Small rank = fresh data = acts first<br/>MIGRATE blocks, split large keys first
Summary
Cluster's complexity lies not in "sharding" but in inconsistency among three tables : slot ownership table converges slowly via Gossip; migration state tables overlap briefly between source and target; client local cache tables are older than anyone. MOVED and ASK are the correction protocols designed for these three tables' time gaps — MOVED corrects permanent errors, ASK tolerates temporary misalignment. Understanding this answers every "why does this command exist" question.
Next up: Sharding solved capacity, but memory is finite — why don't expired keys disappear immediately, and who gets evicted when memory is full? "Expiration Deletion & Memory Eviction — The Trade-offs Behind Eight Eviction Policies" will cover lazy + periodic deletion dual mechanism, LRU clock approximation, LFU probabilistic counters, and applicability of all 8 maxmemory-policy options.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
