Redis High Availability: Master-Slave, Sentinel & Cluster Deep Dive with Production Code
This article comprehensively covers Redis high availability architectures — master-slave replication, Sentinel automatic failover, and Redis Cluster sharding — with detailed working principles, pros/cons, comparison tables, production-grade Java code for Jedis and Lettuce clients, hash-tag routing for cross-slot operations, Spring Retry integration, and solutions for split-brain, async replication data loss, failover downtime, data skew, and cluster scaling latency.
Redis High Availability Solutions Overview
This article provides a thorough examination of three core Redis high availability architectures: master-slave replication, Sentinel mode, and Redis Cluster, along with enterprise optimizations, cloud-native approaches, and production-ready Java implementations.
1. Master-Slave Replication (Foundation)
Master-slave replication is the foundation of all HA solutions, solving data backup and read-write separation .
Core Architecture
Working Principle
Replica starts and sends PSYNC command to master
First connection: master performs full replication (generates RDB file)
Subsequent connections: master performs incremental replication (sends commands after offset)
Pros & Cons
✅ Pros: Simple implementation, linear read scaling, data backup
❌ Cons: No automatic failover (manual intervention required), single-master write bottleneck
Applicable Scenarios
Test environments, low-traffic services (QPS < 1000)
Scenarios needing only read-write separation and data backup
2. Sentinel Mode (Automatic Failover)
Built on master-slave replication, solves automatic failover — most common for small-to-medium production.
Three Core Functions
Monitoring : Continuously checks master/replica health
Leader Election : Automatically elects new master from replicas on failure
Notification : Publishes failover results to clients and other nodes
Core Architecture
Failover Process
Single sentinel detects master unreachable → Subjective Down (SDOWN)
Quorum of sentinels confirm → Objective Down (ODOWN)
Sentinel cluster elects a leader sentinel
Leader elects new master from healthy replicas
Updates config, notifies clients, old master becomes replica
Pros & Cons
✅ Pros: Automatic failover, high availability, simple operations
❌ Cons: No horizontal write scaling , single-master write bottleneck; sentinel cluster itself needs HA
Applicable Scenarios
Small-to-medium production (QPS < 10000)
Low write pressure, high read pressure workloads
3. Redis Cluster (Sharded HA)
Solves single-node write bottleneck and horizontal scaling — standard for high-traffic production.
Core Concept: 16384 Hash Slots
All data mapped to 16384 hash slots
Each master owns a subset of slots
Client computes slot via CRC16(key) % 16384 and connects directly
Core Architecture
Failover Mechanism
Each node monitors others via Gossip protocol
On master failure, its replica auto-promotes via election by other masters
Cluster updates slot mapping; clients follow MOVED redirection
Pros & Cons
✅ Pros: Linear horizontal scaling for both reads and writes; automatic failover
❌ Cons: Operational complexity; no cross-slot multi-key ops (MGET, MSET); brief impact during data migration
Applicable Scenarios
High-traffic, high-concurrency production (QPS > 10000)
Data volume exceeds single-node memory capacity
Comparison of Three Solutions
Master-Slave : Min nodes 2; Failover: Manual; Horizontal scaling: No; Write capacity: Single-master bottleneck; Read capacity: Linear scaling; Ops complexity: Low; Best for: Test, low traffic
Sentinel : Min nodes 5 (3 sentinels + 2 master-slave); Failover: Auto; Horizontal scaling: No; Write capacity: Single-master bottleneck; Read capacity: Linear scaling; Ops complexity: Medium; Best for: Small-medium production
Redis Cluster : Min nodes 6 (3 masters + 3 replicas); Failover: Auto; Horizontal scaling: Linear; Write capacity: Linear scaling; Read capacity: Linear scaling; Ops complexity: High; Best for: High traffic, big data
Enterprise Optimizations & Cloud-Native
Read-write separation optimization : Client-side sharding or proxies (Twemproxy, Codis)
Persistence strategy : Master disables RDB; replicas enable RDB+AOF hybrid for performance/safety balance
Slow query monitoring : Regular cleanup, reasonable timeouts
Cloud-native : Redis Operator on Kubernetes (e.g., Redis Cluster Operator) for automated deploy, scaling, failover
Managed cloud services : Alibaba Cloud Redis, Tencent Cloud Redis for core business (99.99%+ availability)
Interview-Driven Selection Strategy
Early project: Master-slave + Sentinel — simple ops, low cost
Growth phase: Migrate to Redis Cluster when single-master write bottlenecks
Cloud-native env: Prefer Redis Operator for automation
Core business: Use managed cloud Redis for 99.99%+ SLA
Core Code & Technical Highlights
1. Jedis Sentinel Connection (Production Config)
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;
import java.util.HashSet;
import java.util.Set;
public class RedisSentinelConfig {
// Highlight 1: Singleton + double-checked locking for unique pool
private static volatile JedisSentinelPool jedisSentinelPool;
public static JedisSentinelPool getJedisSentinelPool() {
if (jedisSentinelPool == null) {
synchronized (RedisSentinelConfig.class) {
if (jedisSentinelPool == null) {
// Sentinel node addresses
Set<String> sentinels = new HashSet<>();
sentinels.add("192.168.1.10:26379");
sentinels.add("192.168.1.11:26379");
sentinels.add("192.168.1.12:26379");
// Highlight 2: Fine-grained pool config to prevent leaks
jedisSentinelPool = new JedisSentinelPool(
"mymaster", // master name
sentinels,
"password123", // Redis password
2000, // connection timeout
100, // max connections
10, // min idle connections
30000 // max idle time
);
}
}
}
return jedisSentinelPool;
}
public static void main(String[] args) {
try (Jedis jedis = getJedisSentinelPool().getResource()) {
jedis.set("key", "value");
System.out.println(jedis.get("key"));
}
}
}2. Spring Boot + Redis Cluster (Lettuce Client)
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.StringRedisSerializer;
@Configuration
public class RedisClusterConfig {
@Bean
public LettuceConnectionFactory redisConnectionFactory() {
RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration();
// Cluster node addresses
clusterConfig.clusterNode("192.168.1.20", 6379);
clusterConfig.clusterNode("192.168.1.21", 6379);
clusterConfig.clusterNode("192.168.1.22", 6379);
clusterConfig.setPassword("password123");
// Highlight 3: Enable auto-follow on cluster redirects
clusterConfig.setMaxRedirects(3);
return new LettuceConnectionFactory(clusterConfig);
}
@Bean
public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory connectionFactory) {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory);
// Highlight 4: String serializer to avoid JDK serialization garbage
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(new StringRedisSerializer());
template.afterPropertiesSet();
return template;
}
}3. Custom Hash Tag Routing for Cross-Slot Operations
Solves Redis Cluster's lack of cross-slot multi-key support.
/**
* Highlight: Custom hash tag utility
* Forces related keys into same hash slot for MGET/MSET support
*/
public class RedisHashTagUtil {
private static final String HASH_TAG_START = "{";
private static final String HASH_TAG_END = "}";
/**
* Generate key with hash tag
* @param businessId Business ID (user ID, order ID)
* @param keySuffix Key suffix
* @return Key with hash tag
*/
public static String generateKey(String businessId, String keySuffix) {
return HASH_TAG_START + businessId + HASH_TAG_END + ":" + keySuffix;
}
public static void main(String[] args) {
// All keys for same user route to same slot
String userInfoKey = generateKey("user123", "info");
String userOrderKey = generateKey("user123", "orders");
String userCartKey = generateKey("user123", "cart");
// Now safe to execute MGET
// redisTemplate.opsForValue().multiGet(Arrays.asList(userInfoKey, userOrderKey, userCartKey));
}
}4. Client Retry on Master-Replica Failover
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Component;
import redis.clients.jedis.exceptions.JedisConnectionException;
@Component
public class RedisRetryService {
/**
* Highlight: Spring Retry for auto-retry during failover
* On connection exception, retry up to 3 times with 1s base delay, exponential backoff
*/
@Retryable(
value = JedisConnectionException.class,
maxAttempts = 3,
backoff = @Backoff(delay = 1000, multiplier = 2)
)
public String getWithRetry(String key) {
return redisTemplate.opsForValue().get(key);
}
}Technical Challenges & Solutions Summary
Master-Replica Data Inconsistency : Replica data stale, reads return dirty data. Root cause: Replication lag; network jitter breaks incremental sync. Solution: 1. Monitor lag via info replication lag field 2. Force read-master for critical biz 3. Enable repl-disable-tcp-nodelay to reduce latency
Sentinel Split-Brain : Network partition creates two masters, data loss. Root cause: Sentinel quorum unreachable; old master accepts writes after recovery. Solution: 1. Set min-replicas-to-write 1 (require ≥1 replica ack) 2. Set min-replicas-max-lag 10 (reject if lag >10s) 3. Odd sentinel count ≥3
Cluster Cross-Slot Ops : MGET/MSET /transactions throw CrossSlot error. Root cause: Keys map to different slots; cluster forbids cross-slot. Solution: 1. Hash tags to co-locate keys (recommended) 2. Client-side split/merge 3. Proxy layer (Codis) for auto handling
Large Cluster Data Migration : Scale in/out migration slow, impacts business. Root cause: Full RDB transfer large; migration blocks node. Solution: 1. Incremental migration: full then incremental sync 2. Off-peak execution 3. Rate-limit via cluster-migration-barrier Failover Request Loss : Writes not synced to replica lost on master failure. Root cause: Async replication: master returns success before replica ack. Solution: 1. WAIT command for N replica acks 2. Business idempotency for retries 3. Redis 7.0 WAITAOF for strong consistency
Cache Avalanche : Mass expiry spikes DB load. Root cause: Uniform TTL; cluster-wide outage. Solution: 1. Random TTL jitter (±5 min) 2. HA cluster 3. Degradation & circuit breaking
Interview Transcript Highlights
Split-Brain Deep Dive
Network partition isolates old master; sentinel elects new master. Both accept writes. On partition heal, old master data overwritten. Solution: min-replicas-to-write 1 + min-replicas-max-lag 10 — old master refuses writes when isolated, sacrificing availability for consistency.
Async Replication Data Loss
Use WAIT for strong consistency (financial inventory) accepting latency; otherwise accept loss, rely on AOF appendfsync everysec and application-level eventual consistency compensation.
Cluster Client Routing
Smart clients (Lettuce, JedisCluster) cache slot-map. On MOVED (slot migrated) or ASK (migrating), client auto-refreshes topology and retries — transparent to business.
Cluster Limitations
Multi-key ops require same slot. Use {} hash tags to co-locate, but avoid abuse. Cross-node transactions unsupported — design keys to avoid.
Java Client Integration Details
Sentinel Mode: Auto-Discovery & Reconnect
@Configuration
public class RedisSentinelConfig {
@Bean
public LettuceConnectionFactory lettuceConnectionFactory() {
RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
.master("mymaster")
.sentinel("192.168.1.10", 26379)
.sentinel("192.168.1.11", 26379)
.sentinel("192.168.1.12", 26379);
sentinelConfig.setPassword(RedisPassword.of("sentinel_pass"));
LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
.readFrom(ReadFrom.REPLICA_PREFERRED) // Read from replica first
.commandTimeout(Duration.ofSeconds(2)) // Short timeout for failover
.build();
return new LettuceConnectionFactory(sentinelConfig, clientConfig);
}
@Bean
public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory factory) {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(factory);
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(new GenericJackson2JsonRedisSerializer());
return template;
}
}Highlights: ReadFrom.REPLICA_PREFERRED for read scaling; client auto-subscribes to sentinel pub/sub for 1-3s topology refresh; short commandTimeout prevents hang during failover.
Cluster Mode: Smart Redirect & Topology Refresh
@Configuration
public class RedisClusterConfig {
@Bean
public LettuceConnectionFactory clusterConnectionFactory() {
RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(Arrays.asList(
"192.168.2.10:6379",
"192.168.2.11:6379",
"192.168.2.12:6379"
));
clusterConfig.setPassword("cluster_pass");
// Production must: adaptive topology refresh
ClusterTopologyRefreshOptions refreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh(Duration.ofSeconds(30)) // Periodic check
.enableAllAdaptiveRefreshTriggers() // Immediate refresh on MOVED/ASK/role change
.build();
LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
.clientOptions(ClusterClientOptions.builder()
.topologyRefreshOptions(refreshOptions)
.build())
.build();
return new LettuceConnectionFactory(clusterConfig, clientConfig);
}
}Highlights: ClusterTopologyRefreshOptions prevents stale mappings during migration; enableAllAdaptiveRefreshTriggers() + 30s periodic = dual safety; connection pool handles per-shard master-replica failover transparently.
Business-Layer Defensive Coding: Degradation
@Service
public class CacheService {
@Autowired
private RedisTemplate<String, Object> redisTemplate;
public Optional<User> getUserById(String id) {
try {
User user = (User) redisTemplate.opsForValue().get("user:" + id);
return Optional.ofNullable(user);
} catch (RedisConnectionFailureException e) {
log.warn("Redis unavailable, fallback to DB", e);
return Optional.ofNullable(userDao.findById(id));
} catch (Exception e) {
log.error("Redis error", e);
return Optional.empty();
}
}
}Highlights: Catch RedisConnectionFailureException (Lettuce throws on disconnect/cluster down) to prevent cache avalanche; DB fallback with monitoring; combine with Hystrix/Sentinel circuit breaker for system protection.
Four Critical Production Pitfalls & Fixes
🧠 Split-Brain : Network partition → two masters → silent data loss on heal. Solution: ① min-replicas-to-write + min-replicas-max-lag on master to reject writes when isolated ② Client alerts + manual recovery
📉 Async Replication Loss : Writes in flight lost on master crash. Solution: ① WAIT for critical biz (finance) — sync to N replicas before ack ② AOF appendfsync everysec for disk durability
🔄 Failover Downtime (2-10s) : All writes fail during sentinel election + promotion. Solution: ① Client-side retry with exponential backoff (Lettuce default) ② Local cache (Caffeine) for read tolerance ③ Tune sentinel down-after-milliseconds not too high
⚡ Data Skew & Hot Keys : Some slots overloaded in Cluster. Solution: ① Hot key multi-replica scatter: hot:key:{hash} random local pick ② Monitor & redesign keys with {} to co-locate related keys, avoid big-key clustering ③ Upgrade shard spec or vertical scale
🌀 Cluster Resharding Latency Spikes : ASK redirects during migration cause jitter. Solution: ① Enable adaptive topology refresh to reduce repeated ASK ② Batch, off-peak migration; throttle migrate pipeline speed ③ For latency-sensitive biz, temporarily read from replicas during migration
Most Overlooked Yet Fatal Pitfall
Split-brain . Teams assume auto-failover = safety. Without min-replicas-to-write, split-brain silently loses all writes during partition — undetectable until reconciliation. Mandatory in our production: all Redis masters must have anti-split-brain params or deployment blocked.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
