Redis Persistence Deep Dive: From a Major P0 Outage to RDB+AOF Hybrid Implementation

1. Incident Root Cause

During a flash‑sale the host of the primary Redis node lost power. Sentinel promoted a replica that contained almost no data, causing:

Lost inventory reservation keys → incorrect stock counts

Lost user sessions → massive re‑logins

Rate‑limit counters reset → traffic burst to downstream services

Cache miss cascade → MySQL CPU spikes, connection‑pool exhaustion, request queueing and time‑outs

Three underlying problems made the outage a P0 incident:

Redis was used as a pure cache while actually storing critical business state.

Master‑slave replication was mistaken for durable persistence.

No disaster‑recovery drills existed; only configuration was present.

Conclusion: replication provides high availability, not data durability.

2. Redefining Redis’s Role

When Redis stores login tokens, distributed locks, pre‑reserved inventory, leaderboards, rate‑limit windows, delayed‑task state, Stream messages, or idempotency flags, the data cannot be discarded like a page cache. The design goal shifts from “as fast as possible” to “as fast as possible while being recoverable”.

3. Case Study – Inventory Reservation

3.1 Business Keys

stock:sku:{skuId}

– available inventory reserve:order:{orderId} – reservation record idempotent:createOrder:{requestId} – idempotency flag user:session:{token} – login session stream:order_timeout – delayed‑order stream

3.2 Scale Characteristics

Peak QPS: tens of thousands

Instance memory: several GB to tens GB

Hot SKU writes are highly concurrent

Recovery must not lose inventory or reservation state

3.3 Problems with the Original Setup

Typical naïve configuration:

Redis master‑slave + Sentinel

Replica treated as a backup

AOF disabled

RDB interval too long or disabled

No backup to object storage

This works only while the machine stays up. Failure scenarios that expose data loss include host power‑off, cloud‑disk jitter corrupting AOF tails, large‑memory fork pauses, replication lag during failover, and container recreation without a persistent volume.

4. Persistence Mechanisms

4.1 RDB – Point‑in‑Time Snapshots

When BGSAVE runs, Redis forks a child process that serialises the current memory to disk while the parent continues serving requests. Advantages: compact file, fast load, suitable for cold backups. Drawbacks: data between snapshots cannot be recovered, the fork blocks the main thread, and Copy‑On‑Write (COW) can inflate memory during heavy writes.

4.2 AOF – Write‑Ahead Log

Each write command is appended to an in‑memory buffer and flushed to the kernel page cache according to appendfsync policy: always – every command is fsynced immediately (lowest data risk, highest latency) everysec – buffered writes are flushed once per second (up to 1 s data loss, balanced performance, default for most production workloads) no – relies on OS flushing (uncontrolled loss window, lowest latency, only for pure cache)

4.3 AOF Rewrite

A rewrite forks a child that scans the current dataset and writes a minimal set of commands that reconstruct the same state, then atomically replaces the old AOF. The result is a shorter AOF and faster restart, but the fork and disk I/O still cause pauses.

4.4 Hybrid Persistence (RDB + AOF)

Since Redis 4.0, setting aof-use-rdb-preamble yes stores an RDB snapshot in the first part of the AOF file, followed by incremental AOF entries. On restart most data loads quickly from the RDB portion, then recent writes are replayed from the AOF tail. This combines fast recovery with data completeness and is the recommended default for most stateful workloads.

5. Persistence ≠ High Availability

5.1 What Replication Solves

Read scaling

Fail‑over prerequisite

Node‑level HA

5.2 What Replication Does NOT Guarantee

Last‑minute writes are always replicated

Replica data is persisted locally

Erroneous writes are not rolled back

Logical corruption is not filtered

Application‑level WAIT numreplicas timeout can increase write‑ack confidence but cannot replace AOF/RDB durability.

6. Engineering Decisions per Business Type

Pure cache, fully rebuildable – disable persistence or keep low‑frequency RDB snapshots (max performance). Do not mix session or inventory keys.

Session, rate‑limit, idempotency – enable AOF everysec plus hybrid persistence (tiny loss window). Do not rely on RDB only.

Leaderboard / hot profile data – use RDB + AOF everysec (balances speed and real‑time updates). Avoid slave‑only setups.

Redis Stream queues – enable AOF everysec; optionally dual‑write to a message queue. Do not treat Redis as a strong message system.

Flash‑sale inventory & order reservation – AOF everysec + hybrid + regular backups + DR drills (loss directly breaks business constraints). Avoid pure‑cache mindset.

Financial core state – do NOT rely solely on native Redis (it is not a strongly consistent ledger). Use a relational database or event store for the authoritative facts.

7. Recommended Production Architecture

A four‑layer safeguard:

Runtime HA: master + replicas with Sentinel or Redis Cluster.

Real‑time recovery: AOF (prefer everysec).

Point‑in‑time rollback: periodic RDB snapshots.

Cross‑node disaster recovery: backup files to object storage.

Typical data flow: Application cluster → Redis master → Replicas A & B → AOF + hybrid → Periodic RDB → Object storage / persistent volume → Monitoring & alerting → DR drills.

8. Core Order‑Reservation Flow

8.1 Normal Order Path

createOrder(requestId, skuId, count)
SETNX idempotent:createOrder:{requestId}
EVAL reserve_stock.lua
-- success → write draft order → emit order‑created event → return orderId

8.2 Async Confirmation & Compensation

On payment success the order is finalised; on timeout a compensating Redis command releases the reserved stock.

8.3 Recovery After Redis Failure

Detect node failure.

Load local AOF/RDB; if unavailable pull the latest backup from object storage.

Replay business compensation events.

Validate inventory, reservation and idempotency state.

Gray‑release traffic.

Switch to full traffic.

9. Production‑Grade Configuration (redis.conf)

bind 0.0.0.0
port 6379
protected-mode yes
dir /data
dbfilename dump.rdb
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 512mb
aof-use-rdb-preamble yes
maxmemory 12gb
maxmemory-policy noeviction
repl-backlog-size 256mb
client-output-buffer-limit slave 512mb 128mb 60
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes

9.1 Common Pitfalls

maxmemory-policy

must not evict business keys for stateful instances. stop-writes-on-bgsave-error should stay enabled to avoid silent data loss. appendfsync everysec balances durability and latency; switching to always requires careful disk‑latency assessment. no-appendfsync-on-rewrite yes reduces I/O pressure during AOF rewrite but widens the loss window.

10. System‑Level Optimisations

10.1 Fork‑Induced Latency

During BGSAVE or BGREWRITEAOF, the fork blocks the single Redis thread. Large‑memory instances experience noticeable pauses, leading to request queuing, response jitter and upstream retries.

10.2 Transparent Huge Pages (THP)

THP increases the cost of COW. Production systems should disable it, e.g. echo never > /sys/kernel/mm/transparent_hugepage/enabled.

10.3 Memory Headroom for COW

Reserve space for Redis process, COW copies, OS page cache and side‑car agents; otherwise a fork can cause OOM on a busy instance.

10.4 Disk Selection

Prioritise low fsync latency, high sequential write throughput, stable QoS under burst, and sufficient capacity for AOF growth. A slow disk can turn the everysec flush into an implicit block.

11. Container & Kubernetes Deployment

11.1 Docker‑Compose Example

version: "3.8"
services:
  redis:
    image: redis:7.2-alpine
    container_name: redis-state
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - ./data:/data
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5

Key points: mount a persistent /data volume and ensure the dir directive in the config points to that volume.

11.2 StatefulSet Example

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-state
spec:
  serviceName: redis-state
  replicas: 1
  selector:
    matchLabels:
      app: redis-state
  template:
    metadata:
      labels:
        app: redis-state
    spec:
      containers:
        - name: redis
          image: redis:7.2
          command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath: /usr/local/etc/redis/redis.conf
              subPath: redis.conf
          livenessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 20
            periodSeconds: 10
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 5
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

Common K8s mistakes: using a Deployment for stateful Redis, forgetting the PVC, or writing persistence files to the container’s read‑only layer.

12. Recovery Procedures (Not Just Restart)

12.1 AOF Corruption Check

redis-check-aof --fix /data/appendonly.aof

The command truncates the corrupted tail; back up the file first.

12.2 RDB Validation

redis-check-rdb /data/dump.rdb

12.3 Backup Script to Object Storage

#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/data"
TS=$(date +%F-%H%M%S)
TARGET="/backup/redis/${TS}"
mkdir -p "${TARGET}"
cp "${BACKUP_DIR}/dump.rdb" "${TARGET}/dump.rdb"
if [ -f "${BACKUP_DIR}/appendonly.aof" ]; then
  cp "${BACKUP_DIR}/appendonly.aof" "${TARGET}/appendonly.aof"
fi
echo "backup completed: ${TARGET}"

12.4 Standard Recovery Flow

Determine whether the failure is logical corruption or node loss.

If logical, freeze writes before switching master.

Attempt local AOF/RDB recovery first.

If unavailable, pull the latest backup from object storage.

After load, perform business‑level validation (inventory, session, idempotency).

Gray‑release traffic, then switch to full traffic.

13. Observability – Metrics & Alerts

13.1 Essential Metrics

latest_fork_usec

rdb_last_bgsave_status

rdb_last_save_time

aof_enabled

aof_last_bgrewrite_status

aof_last_write_status

aof_current_size

aof_base_size

mem_fragmentation_ratio

used_memory_peak

master_repl_offset

Replica replication lag

13.2 Minimum Alerts

Recent RDB failure

Recent AOF rewrite failure

AOF write errors

Elevated latest_fork_usec Memory approaching the configured limit

Insufficient disk space on the persistent volume

Growing replication lag

14. Performance & Capacity Validation

14.1 Baseline Throughput Tests

Compare four configurations: no persistence, RDB only, AOF everysec, and hybrid. Measure QPS, P95/P99 latency, fsync latency, CPU and I/O utilisation.

14.2 Fork‑Induced Jitter Tests

Run continuous writes while triggering BGSAVE and BGREWRITEAOF. Observe latest_fork_usec, latency spikes and any application time‑outs.

14.3 Failure‑Recovery Tests

Simulate host kill, disk full, or AOF tail corruption; measure recovery time, data‑loss window and business‑level consistency after restore.

14.4 Business Consistency Checks

After recovery verify that Redis‑reserved inventory matches MySQL order facts and that compensation events have been replayed correctly.

15. Common Misconceptions

Replica ≠ backup – errors replicate as well.

RDB alone is insufficient for stateful data.

AOF is not magically safe; it can suffer disk jitter, tail corruption, rewrite failures and a loss window.

All Redis instances should not share the same config; tiered instances need tailored settings.

Recovery is more than a process restart; it must restore business constraints and keep the request flow functional.

16. When to Look Beyond Redis

Consider alternative systems if any of the following hold:

Dataset so large that fork pauses are unacceptable.

Strict strong‑consistency requirements.

Message reliability needs exceed cache semantics.

Memory cost per node becomes prohibitive.

Recovery‑time objectives tighter than AOF/RDB can provide.

Alternatives include persisting core facts in a relational database or event store, using a dedicated message queue for reliable streams, or off‑loading massive cold data to RocksDB‑like storage.

17. Immediate Action Checklist (8 Steps)

Identify all Redis keys whose loss would break business logic.

Separate pure‑cache instances from stateful instances.

Enable AOF everysec plus hybrid persistence on stateful nodes.

Keep periodic RDB snapshots and back them up to object storage.

Verify containers/K8s actually mount persistent volumes for /data and that dir points to them.

Configure alerts for latest_fork_usec, AOF/RDB status, and disk space.

Develop at least one automated recovery script and a business‑validation checklist.

Add compensation and replay mechanisms for inventory, idempotency and session flows.

18. Final Takeaway

Redis persistence is not about memorising the terms RDB, AOF or everysec; it is about recognising three engineering facts:

Performance always trades off with recoverability; the business must decide the acceptable risk.

Durability requires a holistic system of replication, disk persistence, backup, recovery, compensation and monitoring.

Once Redis holds business state it must be treated as a reliable data system rather than a temporary cache.

For most instances that store business state, the practical starting point is:

AOF everysec + hybrid persistence + periodic RDB + object‑storage backup + recovery drills

.