Skip to content
Redis High Availability: Master-Slave, Sentinel & Cluster Deep Dive with Production Code
Databases 29 min read

Redis High Availability: Master-Slave, Sentinel & Cluster Deep Dive with Production Code

This article comprehensively covers Redis high availability architectures — master-slave replication, Sentinel automatic failover, and Redis Cluster sharding — with detailed working principles, pros/cons, comparison tables, production-grade Java code for Jedis and Lettuce clients, hash-tag routing for cross-slot operations, Spring Retry integration, and solutions for split-brain, async replication data loss, failover downtime, data skew, and cluster scaling latency.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Redis High Availability Solutions Overview

This article provides a thorough examination of three core Redis high availability architectures: master-slave replication, Sentinel mode, and Redis Cluster, along with enterprise optimizations, cloud-native approaches, and production-ready Java implementations.

1. Master-Slave Replication (Foundation)

Master-slave replication is the foundation of all HA solutions, solving data backup and read-write separation .

Core Architecture

Master-Slave Architecture
Master-Slave Architecture

Working Principle

Replica starts and sends PSYNC command to master

First connection: master performs full replication (generates RDB file)

Subsequent connections: master performs incremental replication (sends commands after offset)

Pros & Cons

✅ Pros: Simple implementation, linear read scaling, data backup

❌ Cons: No automatic failover (manual intervention required), single-master write bottleneck

Applicable Scenarios

Test environments, low-traffic services (QPS < 1000)

Scenarios needing only read-write separation and data backup

2. Sentinel Mode (Automatic Failover)

Built on master-slave replication, solves automatic failover — most common for small-to-medium production.

Three Core Functions

Monitoring : Continuously checks master/replica health

Leader Election : Automatically elects new master from replicas on failure

Notification : Publishes failover results to clients and other nodes

Core Architecture

Sentinel Architecture
Sentinel Architecture

Failover Process

Single sentinel detects master unreachable → Subjective Down (SDOWN)

Quorum of sentinels confirm → Objective Down (ODOWN)

Sentinel cluster elects a leader sentinel

Leader elects new master from healthy replicas

Updates config, notifies clients, old master becomes replica

Pros & Cons

✅ Pros: Automatic failover, high availability, simple operations

❌ Cons: No horizontal write scaling , single-master write bottleneck; sentinel cluster itself needs HA

Applicable Scenarios

Small-to-medium production (QPS < 10000)

Low write pressure, high read pressure workloads

3. Redis Cluster (Sharded HA)

Solves single-node write bottleneck and horizontal scaling — standard for high-traffic production.

Core Concept: 16384 Hash Slots

All data mapped to 16384 hash slots

Each master owns a subset of slots

Client computes slot via CRC16(key) % 16384 and connects directly

Core Architecture

Cluster Architecture
Cluster Architecture

Failover Mechanism

Each node monitors others via Gossip protocol

On master failure, its replica auto-promotes via election by other masters

Cluster updates slot mapping; clients follow MOVED redirection

Pros & Cons

✅ Pros: Linear horizontal scaling for both reads and writes; automatic failover

❌ Cons: Operational complexity; no cross-slot multi-key ops (MGET, MSET); brief impact during data migration

Applicable Scenarios

High-traffic, high-concurrency production (QPS > 10000)

Data volume exceeds single-node memory capacity

Comparison of Three Solutions

Master-Slave : Min nodes 2; Failover: Manual; Horizontal scaling: No; Write capacity: Single-master bottleneck; Read capacity: Linear scaling; Ops complexity: Low; Best for: Test, low traffic

Sentinel : Min nodes 5 (3 sentinels + 2 master-slave); Failover: Auto; Horizontal scaling: No; Write capacity: Single-master bottleneck; Read capacity: Linear scaling; Ops complexity: Medium; Best for: Small-medium production

Redis Cluster : Min nodes 6 (3 masters + 3 replicas); Failover: Auto; Horizontal scaling: Linear; Write capacity: Linear scaling; Read capacity: Linear scaling; Ops complexity: High; Best for: High traffic, big data

Enterprise Optimizations & Cloud-Native

Read-write separation optimization : Client-side sharding or proxies (Twemproxy, Codis)

Persistence strategy : Master disables RDB; replicas enable RDB+AOF hybrid for performance/safety balance

Slow query monitoring : Regular cleanup, reasonable timeouts

Cloud-native : Redis Operator on Kubernetes (e.g., Redis Cluster Operator) for automated deploy, scaling, failover

Managed cloud services : Alibaba Cloud Redis, Tencent Cloud Redis for core business (99.99%+ availability)

Interview-Driven Selection Strategy

Early project: Master-slave + Sentinel — simple ops, low cost

Growth phase: Migrate to Redis Cluster when single-master write bottlenecks

Cloud-native env: Prefer Redis Operator for automation

Core business: Use managed cloud Redis for 99.99%+ SLA

Core Code & Technical Highlights

1. Jedis Sentinel Connection (Production Config)

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;
import java.util.HashSet;
import java.util.Set;

public class RedisSentinelConfig {
    // Highlight 1: Singleton + double-checked locking for unique pool
    private static volatile JedisSentinelPool jedisSentinelPool;

    public static JedisSentinelPool getJedisSentinelPool() {
        if (jedisSentinelPool == null) {
            synchronized (RedisSentinelConfig.class) {
                if (jedisSentinelPool == null) {
                    // Sentinel node addresses
                    Set<String> sentinels = new HashSet<>();
                    sentinels.add("192.168.1.10:26379");
                    sentinels.add("192.168.1.11:26379");
                    sentinels.add("192.168.1.12:26379");

                    // Highlight 2: Fine-grained pool config to prevent leaks
                    jedisSentinelPool = new JedisSentinelPool(
                        "mymaster", // master name
                        sentinels,
                        "password123", // Redis password
                        2000, // connection timeout
                        100, // max connections
                        10, // min idle connections
                        30000 // max idle time
                    );
                }
            }
        }
        return jedisSentinelPool;
    }

    public static void main(String[] args) {
        try (Jedis jedis = getJedisSentinelPool().getResource()) {
            jedis.set("key", "value");
            System.out.println(jedis.get("key"));
        }
    }
}

2. Spring Boot + Redis Cluster (Lettuce Client)

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.StringRedisSerializer;

@Configuration
public class RedisClusterConfig {
    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {
        RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration();
        // Cluster node addresses
        clusterConfig.clusterNode("192.168.1.20", 6379);
        clusterConfig.clusterNode("192.168.1.21", 6379);
        clusterConfig.clusterNode("192.168.1.22", 6379);
        clusterConfig.setPassword("password123");
        // Highlight 3: Enable auto-follow on cluster redirects
        clusterConfig.setMaxRedirects(3);
        return new LettuceConnectionFactory(clusterConfig);
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory connectionFactory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(connectionFactory);
        // Highlight 4: String serializer to avoid JDK serialization garbage
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(new StringRedisSerializer());
        template.afterPropertiesSet();
        return template;
    }
}

3. Custom Hash Tag Routing for Cross-Slot Operations

Solves Redis Cluster's lack of cross-slot multi-key support.

/**
 * Highlight: Custom hash tag utility
 * Forces related keys into same hash slot for MGET/MSET support
 */
public class RedisHashTagUtil {
    private static final String HASH_TAG_START = "{";
    private static final String HASH_TAG_END = "}";

    /**
     * Generate key with hash tag
     * @param businessId Business ID (user ID, order ID)
     * @param keySuffix Key suffix
     * @return Key with hash tag
     */
    public static String generateKey(String businessId, String keySuffix) {
        return HASH_TAG_START + businessId + HASH_TAG_END + ":" + keySuffix;
    }

    public static void main(String[] args) {
        // All keys for same user route to same slot
        String userInfoKey = generateKey("user123", "info");
        String userOrderKey = generateKey("user123", "orders");
        String userCartKey = generateKey("user123", "cart");
        // Now safe to execute MGET
        // redisTemplate.opsForValue().multiGet(Arrays.asList(userInfoKey, userOrderKey, userCartKey));
    }
}

4. Client Retry on Master-Replica Failover

import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Component;
import redis.clients.jedis.exceptions.JedisConnectionException;

@Component
public class RedisRetryService {
    /**
     * Highlight: Spring Retry for auto-retry during failover
     * On connection exception, retry up to 3 times with 1s base delay, exponential backoff
     */
    @Retryable(
        value = JedisConnectionException.class,
        maxAttempts = 3,
        backoff = @Backoff(delay = 1000, multiplier = 2)
    )
    public String getWithRetry(String key) {
        return redisTemplate.opsForValue().get(key);
    }
}

Technical Challenges & Solutions Summary

Master-Replica Data Inconsistency : Replica data stale, reads return dirty data. Root cause: Replication lag; network jitter breaks incremental sync. Solution: 1. Monitor lag via info replication lag field 2. Force read-master for critical biz 3. Enable repl-disable-tcp-nodelay to reduce latency

Sentinel Split-Brain : Network partition creates two masters, data loss. Root cause: Sentinel quorum unreachable; old master accepts writes after recovery. Solution: 1. Set min-replicas-to-write 1 (require ≥1 replica ack) 2. Set min-replicas-max-lag 10 (reject if lag >10s) 3. Odd sentinel count ≥3

Cluster Cross-Slot Ops : MGET/MSET /transactions throw CrossSlot error. Root cause: Keys map to different slots; cluster forbids cross-slot. Solution: 1. Hash tags to co-locate keys (recommended) 2. Client-side split/merge 3. Proxy layer (Codis) for auto handling

Large Cluster Data Migration : Scale in/out migration slow, impacts business. Root cause: Full RDB transfer large; migration blocks node. Solution: 1. Incremental migration: full then incremental sync 2. Off-peak execution 3. Rate-limit via cluster-migration-barrier Failover Request Loss : Writes not synced to replica lost on master failure. Root cause: Async replication: master returns success before replica ack. Solution: 1. WAIT command for N replica acks 2. Business idempotency for retries 3. Redis 7.0 WAITAOF for strong consistency

Cache Avalanche : Mass expiry spikes DB load. Root cause: Uniform TTL; cluster-wide outage. Solution: 1. Random TTL jitter (±5 min) 2. HA cluster 3. Degradation & circuit breaking

Interview Transcript Highlights

Split-Brain Deep Dive

Network partition isolates old master; sentinel elects new master. Both accept writes. On partition heal, old master data overwritten. Solution: min-replicas-to-write 1 + min-replicas-max-lag 10 — old master refuses writes when isolated, sacrificing availability for consistency.

Async Replication Data Loss

Use WAIT for strong consistency (financial inventory) accepting latency; otherwise accept loss, rely on AOF appendfsync everysec and application-level eventual consistency compensation.

Cluster Client Routing

Smart clients (Lettuce, JedisCluster) cache slot-map. On MOVED (slot migrated) or ASK (migrating), client auto-refreshes topology and retries — transparent to business.

Cluster Limitations

Multi-key ops require same slot. Use {} hash tags to co-locate, but avoid abuse. Cross-node transactions unsupported — design keys to avoid.

Java Client Integration Details

Sentinel Mode: Auto-Discovery & Reconnect

@Configuration
public class RedisSentinelConfig {
    @Bean
    public LettuceConnectionFactory lettuceConnectionFactory() {
        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration()
            .master("mymaster")
            .sentinel("192.168.1.10", 26379)
            .sentinel("192.168.1.11", 26379)
            .sentinel("192.168.1.12", 26379);
        sentinelConfig.setPassword(RedisPassword.of("sentinel_pass"));

        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
            .readFrom(ReadFrom.REPLICA_PREFERRED) // Read from replica first
            .commandTimeout(Duration.ofSeconds(2)) // Short timeout for failover
            .build();
        return new LettuceConnectionFactory(sentinelConfig, clientConfig);
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory factory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(factory);
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(new GenericJackson2JsonRedisSerializer());
        return template;
    }
}

Highlights: ReadFrom.REPLICA_PREFERRED for read scaling; client auto-subscribes to sentinel pub/sub for 1-3s topology refresh; short commandTimeout prevents hang during failover.

Cluster Mode: Smart Redirect & Topology Refresh

@Configuration
public class RedisClusterConfig {
    @Bean
    public LettuceConnectionFactory clusterConnectionFactory() {
        RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(Arrays.asList(
            "192.168.2.10:6379",
            "192.168.2.11:6379",
            "192.168.2.12:6379"
        ));
        clusterConfig.setPassword("cluster_pass");

        // Production must: adaptive topology refresh
        ClusterTopologyRefreshOptions refreshOptions = ClusterTopologyRefreshOptions.builder()
            .enablePeriodicRefresh(Duration.ofSeconds(30)) // Periodic check
            .enableAllAdaptiveRefreshTriggers() // Immediate refresh on MOVED/ASK/role change
            .build();

        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
            .clientOptions(ClusterClientOptions.builder()
                .topologyRefreshOptions(refreshOptions)
                .build())
            .build();
        return new LettuceConnectionFactory(clusterConfig, clientConfig);
    }
}

Highlights: ClusterTopologyRefreshOptions prevents stale mappings during migration; enableAllAdaptiveRefreshTriggers() + 30s periodic = dual safety; connection pool handles per-shard master-replica failover transparently.

Business-Layer Defensive Coding: Degradation

@Service
public class CacheService {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    public Optional<User> getUserById(String id) {
        try {
            User user = (User) redisTemplate.opsForValue().get("user:" + id);
            return Optional.ofNullable(user);
        } catch (RedisConnectionFailureException e) {
            log.warn("Redis unavailable, fallback to DB", e);
            return Optional.ofNullable(userDao.findById(id));
        } catch (Exception e) {
            log.error("Redis error", e);
            return Optional.empty();
        }
    }
}

Highlights: Catch RedisConnectionFailureException (Lettuce throws on disconnect/cluster down) to prevent cache avalanche; DB fallback with monitoring; combine with Hystrix/Sentinel circuit breaker for system protection.

Four Critical Production Pitfalls & Fixes

🧠 Split-Brain : Network partition → two masters → silent data loss on heal. Solution: ① min-replicas-to-write + min-replicas-max-lag on master to reject writes when isolated ② Client alerts + manual recovery

📉 Async Replication Loss : Writes in flight lost on master crash. Solution: ① WAIT for critical biz (finance) — sync to N replicas before ack ② AOF appendfsync everysec for disk durability

🔄 Failover Downtime (2-10s) : All writes fail during sentinel election + promotion. Solution: ① Client-side retry with exponential backoff (Lettuce default) ② Local cache (Caffeine) for read tolerance ③ Tune sentinel down-after-milliseconds not too high

⚡ Data Skew & Hot Keys : Some slots overloaded in Cluster. Solution: ① Hot key multi-replica scatter: hot:key:{hash} random local pick ② Monitor & redesign keys with {} to co-locate related keys, avoid big-key clustering ③ Upgrade shard spec or vertical scale

🌀 Cluster Resharding Latency Spikes : ASK redirects during migration cause jitter. Solution: ① Enable adaptive topology refresh to reduce repeated ASK ② Batch, off-peak migration; throttle migrate pipeline speed ③ For latency-sensitive biz, temporarily read from replicas during migration

Most Overlooked Yet Fatal Pitfall

Split-brain . Teams assume auto-failover = safety. Without min-replicas-to-write, split-brain silently loses all writes during partition — undetectable until reconciliation. Mandatory in our production: all Redis masters must have anti-split-brain params or deployment blocked.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

High AvailabilityRedisData ConsistencySentinelMaster-Slave ReplicationRedis ClusterSplit-BrainJava Client Configuration
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.