Master‑Slave Replication in Redis: Core Mechanics Explained and Production Deployment

Why Re‑examine Redis Replication

Typical tutorials describe replication as a master‑replica pair where writes go to the master and reads to the replicas, and a failover is possible. In production you must understand the replication state machine, offsets, the replication backlog, and how RDB/AOF interact with the replication link; otherwise the system appears "available" but is not truly reliable.

Business Scenario: E‑commerce Cache Cluster

An e‑commerce platform caches product details, inventory snapshots, marketing configs, and shopping carts using a one‑master‑two‑replica setup. The master handles hot reads, writes, second‑level config pushes, and serves as a fast recovery layer during failures.

What Replication Actually Solves

Read‑Write Split : Distribute read traffic to replicas – higher query throughput – limited by replication lag.

Data Redundancy : Multiple copies of the same data – reduces single‑node failure risk – does not provide strong consistency.

HA Switch Base : Provides candidates for Sentinel/Cluster – supports master failover – replication itself does not auto‑switch.

Disaster Recovery & Offline Reads : Delayed, backup, cross‑region replicas – lower primary risk – higher cross‑region latency.

Replication only "asynchronously propagates" data; it does not guarantee strong consistency or zero loss.

Replication Architecture Overview

The system can be split into four layers:

PSYNC/ACK

RDB snapshots

Incremental command propagation

Application write path

Key components include master_repl_offset, replication backlog, replica output buffers, and the run id / replication id.

Full Sync vs Partial Sync

Initial sync always performs a full sync (BGSAVE → RDB transfer → incremental replay). After a brief disconnection, if the backlog still contains the missing data the replica can continue with CONTINUE (partial sync); otherwise a full FULLRESYNC is required.

In production the goal is not to make full sync fast, but to avoid triggering it.

Why Full Sync Is Expensive

Master thread receives the replica request.

Master forks a child process.

Child generates an RDB snapshot.

Master continues serving writes.

Writes are written to the backlog and replica output buffers.

RDB is sent to the replica.

Replica clears old data and loads the snapshot.

Master replays the accumulated incremental commands.

The BGSAVE step incurs CPU for page‑table copying, memory growth from copy‑on‑write, and latency proportional to data size. If memory is near its limit, fork can cause latency spikes, OOM, RDB failure, or replication interruption.

Backlog Sizing

Formula:

repl-backlog-size >= peak write bytes × tolerated outage time × safety factor

. Example: 8 MB/s peak write, 45 s tolerance, factor 2 → 720 MB backlog.

If the backlog is too small, the replica cannot cover the disconnection window, leading to a replication storm (continuous full syncs).

Replication ID and Offset

The master’s master_replid identifies the current replication history. Replicas store the last replication ID and offset; on reconnection they send these to the master to decide between CONTINUE or FULLRESYNC.

Partial Sync Core Concepts

Replication ID ( master_replid)

Replication offset ( master_repl_offset)

Backlog buffer ( replication backlog)

Asynchronous Nature and Consistency

Writes are acknowledged by the master before being propagated to replicas. If the master crashes before propagation, the writes are lost. Read‑after‑write may see stale data on replicas, so critical paths (order creation, payment confirmation, immediate config verification) must read from the master.

Strengthening Consistency

Force reads from master for strong‑consistency queries.

Use version checks after writes.

Leverage the WAIT command to wait for replica acknowledgments.

Configure min-replicas-to-write and min-replicas-max-lag to block writes when replicas are unhealthy.

Production‑grade Spring Boot + Lettuce Example

Maven Dependencies

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>
    ...
</dependencies>

application.yml

redis:
  topology:
    master-host: redis-master
    master-port: 6379
    replica-nodes:
      - redis-replica-1:6380
      - redis-replica-2:6381
  auth:
    password: ${REDIS_PASSWORD:changeit}
  timeout:
    command-ms: 1500
  pool:
    max-active: 64
    max-idle: 16
    min-idle: 8
    max-wait-ms: 1000

RedisConfig.java

@Configuration
@EnableConfigurationProperties(RedisReplicaProperties.class)
public class RedisConfig {
    @Bean
    public LettuceConnectionFactory redisConnectionFactory(RedisReplicaProperties props) {
        RedisStaticMasterReplicaConfiguration topology =
            new RedisStaticMasterReplicaConfiguration(props.getTopology().getMasterHost(), props.getTopology().getMasterPort());
        topology.setPassword(RedisPassword.of(props.getAuth().getPassword()));
        addReplicaNodes(topology, props.getTopology().getReplicaNodes());
        GenericObjectPoolConfig<?> poolConfig = new GenericObjectPoolConfig<>();
        poolConfig.setMaxTotal(props.getPool().getMaxActive());
        poolConfig.setMaxIdle(props.getPool().getMaxIdle());
        poolConfig.setMinIdle(props.getPool().getMinIdle());
        poolConfig.setMaxWait(Duration.ofMillis(props.getPool().getMaxWaitMs()));
        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
            .readFrom(ReadFrom.REPLICA_PREFERRED)
            .commandTimeout(Duration.ofMillis(props.getTimeout().getCommandMs()))
            .build();
        return new LettuceConnectionFactory(topology, clientConfig);
    }
    // helper method omitted for brevity
}

Static topology works for stable DNS (e.g., StatefulSet). For dynamic topologies switch to Sentinel or Cluster.

Docker‑Compose Minimal Lab

Master Config (redis‑master.conf)

port 6379
bind 0.0.0.0
protected-mode no
requirepass master123
appendonly yes
appendfsync everysec
repl-backlog-size 256mb
repl-backlog-ttl 3600
repl-diskless-sync yes
repl-diskless-sync-delay 5
client-output-buffer-limit replica 256mb 128mb 60
min-replicas-to-write 1
min-replicas-max-lag 10

Replica Configs

port 6380
bind 0.0.0.0
protected-mode no
requirepass master123
masterauth master123
replicaof redis-master 6379
replica-read-only yes

docker‑compose.yml

version: "3.8"
services:
  redis-master:
    image: redis:7.2
    container_name: redis-master
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-master.conf:/usr/local/etc/redis/redis.conf
    ports:
      - "6379:6379"
    networks:
      - redis-net
  redis-replica-1:
    image: redis:7.2
    container_name: redis-replica-1
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-replica-1.conf:/usr/local/etc/redis/redis.conf
    ports:
      - "6380:6380"
    depends_on:
      - redis-master
    networks:
      - redis-net
  redis-replica-2:
    image: redis:7.2
    container_name: redis-replica-2
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-replica-2.conf:/usr/local/etc/redis/redis.conf
    ports:
      - "6381:6381"
    depends_on:
      - redis-master
    networks:
      - redis-net
networks:
  redis-net:
    driver: bridge

Validate with

docker exec -it redis-master redis-cli -a master123 INFO replication

and check role, connected_slaves, master_repl_offset, and replica lag.

Kubernetes Production Deployment

StatefulSet provides stable pod names (e.g., redis-0 as master) and stable DNS, which is essential because replicas must know a fixed master address. A headless Service removes the clusterIP, enabling direct DNS resolution.

Key YAML Snippets

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
        - name: init-config
          image: redis:7.2
          command: ["/bin/sh", "-c"]
          args:
            - |
              if [ "$(hostname)" = "redis-0" ]; then
                cp /conf/master.conf /work/redis.conf
              else
                cp /conf/replica.conf /work/redis.conf
              fi
          volumeMounts:
            - name: config
              mountPath: /conf
            - name: workdir
              mountPath: /work
      containers:
        - name: redis
          image: redis:7.2
          command: ["redis-server", "/work/redis.conf"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: workdir
              mountPath: /work
            - name: data
              mountPath: /data
      volumes:
        - name: config
          configMap:
            name: redis-config
        - name: workdir
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

Replica configuration includes replicaof redis-0.redis 6379 and masterauth ${REDIS_PASSWORD}. StatefulSet guarantees one‑to‑one PVC per pod, stable DNS, and ordered startup, which solves the common container‑IP problem.

High‑Concurrency Engineering Upgrades

Read Traffic Management

Hot‑key pre‑warming.

Local Caffeine cache + Redis second‑level cache.

Hot‑key isolation.

Validate replica lag before big promotions.

Write Traffic Management

Batch writes.

Asynchronous write‑back via MQ.

Resource Isolation

Separate connection pools for reads and writes.

Thread‑pool isolation for cache rebuilds.

Key‑space isolation (different instances or shards).

Dedicated replica roles (read‑only vs backup).

Rate‑Limiting, Circuit‑Breaking, Degradation

Limit non‑critical cache queries.

Fast fallback on Redis timeout.

Avoid blocking business threads on cache failures.

Local cache or default values for hot endpoints.

Data Consistency Design: When to Read from Master

Write‑then‑read scenarios (order status, payment confirmation, inventory check).

Immediate config verification.

Other scenarios (product detail pages, marketing rule queries) can safely read from replicas.

Exception Handling: Disconnection, Retry, Compensation

Replica Reconnection Flow

On disconnection the replica records its replid and offset, attempts reconnection, checks if the backlog can cover the gap, then either continues with incremental sync or triggers a full resync, finally loading the RDB and replaying increments.

Application‑Side Failure Strategies

Cache miss fallback to DB with short‑term throttling.

Switch to master for inventory queries.

Fallback to last local snapshot for config.

Compensation Techniques

Retry failed async cache writes.

Dead‑letter queues.

Manual compensation scripts.

Periodic full‑sync correction jobs.

Performance Bottleneck Analysis

CPU Hotspots on Master

Frequent BGSAVE triggers.

Replication storms.

Large keys causing heavy fork cost.

AOF rewrite overlapping with full sync.

Network Saturation

Multiple replicas full‑sync simultaneously.

Cross‑region replicas.

Diskless sync when network is already the bottleneck.

Replica Lag Causes

Insufficient CPU/memory on replica.

Replica serving heavy read traffic.

Output buffer backlog.

Replica loading RDB.

Memory Risks

Fork overhead.

Copy‑on‑write memory amplification.

Backlog and output buffer consumption.

Large keys and bulk writes.

Observability: Metrics & Alerts

connected_slaves

: current replica count – drop indicates replica failure. master_repl_offset: master replication offset – used to gauge sync speed. slave_repl_offset: replica offset – large gap is dangerous. lag: replica ACK latency – increasing lag means replica falling behind. repl_backlog_size: backlog capacity – too small prevents partial sync. repl_backlog_histlen: used length – near limit signals risk of disconnection. latest_fork_usec: last fork duration – high value indicates costly full sync.

Typical alerts:

Replica lag > 3 s for >1 min.

Growing offset gap. connected_slaves below expected.

Frequent FULLRESYNC events.

Rising fork time.

Common Production Issues & Mitigations

Replication Storm : Tune repl-backlog-size, increase repl-diskless-sync-delay, stagger replica reconnections, consider tree‑shaped replication.

Backlog Too Small : Size based on peak write rate (e.g., 720 MB for 8 MB/s × 45 s × 2).

Slow Replica Dragging Master : Adjust client-output-buffer-limit replica, limit replica query load, monitor replica health.

Master Restart Full‑Sync Avalanche : Avoid restarts during traffic peaks, batch replica recovery.

Container IP Hard‑coding : Use StatefulSet DNS, avoid static replicaof IPs.

Using Replica as Backup : Separate roles; keep backup replicas read‑only and isolated from traffic.

Large Keys : Split huge hashes/lists/zsets, audit historic large keys.

Treating Replication as Full HA : Deploy Sentinel for automatic failover or move to Redis Cluster for sharding.

Technology Selection

Simple Master‑Slave : Single‑shard, read‑heavy, manual ops acceptable – simple, low cost, no auto‑failover.

Master‑Slave + Sentinel : Need automatic master‑failover – auto recovery, still single‑master write bottleneck.

Redis Cluster : Large data, need sharding – horizontal scaling, auto‑sharding, higher complexity, client requirements.

Replication is the transport layer of Redis HA, not the final HA solution.

Practical Checklist

Re‑calculate repl-backlog-size based on write throughput.

Enable replication monitoring and alerts (lag, full‑sync count).

Force master reads for write‑then‑read paths.

Set appropriate replica output buffer limits.

Prefer StatefulSet with stable DNS in container environments.

Audit large keys to avoid fork and full‑sync blow‑up.

Avoid master restarts or topology changes during traffic peaks.

Plan to integrate Sentinel or Cluster after mastering replication.

Conclusion

Redis master‑slave replication is more than a simple "write to master, read from slaves" pattern; it involves a state machine with replication IDs, offsets, a circular backlog, and RDB snapshots. Full sync is costly and should be avoided; partial sync depends on a properly sized backlog. Because replication is asynchronous, engineers must design read routing, configure safety parameters ( min-replicas-to-write, WAIT), isolate resources, and monitor key metrics. Understanding these fundamentals clarifies why Sentinel provides automatic failover and why Cluster enables sharding, making the overall high‑availability architecture more predictable and robust.