Master Redis: Persistence, Caching Pitfalls, Clustering & Performance Tips

Redis Persistence Mechanism

Redis provides two durable persistence options that periodically flush the in‑memory dataset to disk.

RDB (Redis Database Backup) : Redis forks a child process (via fork()) that receives a copy of the parent’s memory. The child writes the snapshot to a temporary file and, once the write completes, atomically replaces the previous dump.rdb file. The snapshot interval is configured with the save directive in redis.conf (e.g., save 900 1 – save if at least one key changed in 900 seconds).

AOF (Append‑Only File) : Every write command received by the server is appended to appendonly.aof. On restart Redis replays the logged commands to rebuild the dataset. The appendfsync policy controls durability (always, everysec, or no).

If both mechanisms are enabled, Redis prefers AOF recovery because it can replay a more complete command log.

Cache‑Related Problems and Mitigations

Cache Avalanche

When many keys share the same TTL, they may expire simultaneously, causing a sudden surge of database queries that can overload the backend.

Mitigation: Stagger TTLs (add random jitter), use distributed locks (e.g., SETNX) or a queue to limit concurrent DB access during bulk expiration.

Cache Penetration

Requests for non‑existent keys bypass the cache and repeatedly hit the database.

Mitigation: Deploy a Bloom filter to pre‑filter impossible keys, or cache empty results with a short TTL (e.g., 5 minutes) to break the query loop.

Cache Stampede (Hot‑Key Miss)

A hot key expires while many threads request it, overwhelming the database.

Mitigation: Use SETNX to create a short‑lived lock before rebuilding the cache; only the lock holder repopulates the key.

Cache Pre‑warming

Load essential data into Redis before the first user request to avoid latency spikes.

Manual refresh page after deployment.

Automatic load on application start for small datasets.

Scheduled jobs that periodically refresh hot data.

Cache Update Strategies

Periodic cleanup: A background task scans and removes expired entries at fixed intervals.

On‑demand refresh: Each request checks the key’s TTL; if expired, the service fetches fresh data from the source and updates the cache.

Cache Degradation

When the system is under heavy load or a non‑critical service fails, return default values or bypass the cache for the affected data to keep core services responsive.

Hot vs. Cold Data

Hot data is accessed frequently (e.g., daily birthday lists, navigation maps) and benefits from caching. Cold data is rarely accessed; it may be evicted quickly and provides little cache value. A rule of thumb: cache data that is read at least twice before it expires.

Memcached vs. Redis

Persistence: Memcached is purely in‑memory; Redis can persist to disk via RDB/AOF.

Data types: Memcached stores only strings; Redis supports strings, lists, sets, sorted sets, hashes, streams, etc.

Architecture: Both use an event‑driven single‑threaded core, but Redis adds optional I/O threads for networking.

Maximum value size: Redis up to 512 MB per key; Memcached ~1 MB.

Replication: Redis offers built‑in master‑slave replication; Memcached does not.

Why Redis Is Fast

All operations are performed on data resident in RAM, giving O(1) access.

A single‑threaded event loop eliminates context switches and lock contention.

Non‑blocking I/O multiplexing (select/epoll/kqueue) allows thousands of concurrent client connections.

Redis Data Types and Typical Use Cases

String : Simple key/value, counters, basic caching.

Hash : Store object fields (e.g., user session data) allowing field‑level updates.

List : Queues, message streams, pagination via LRANGE.

Set : Global deduplication, set operations (union, intersection, difference).

Sorted Set (ZSET) : Leaderboards, time‑sorted feeds, top‑N queries using scores.

Internal Data Structures

dict

: Hash table that maps keys to values. sds (Simple Dynamic String): Stores binary data with explicit length. skiplist: Used by sorted sets to provide O(log N) range queries. quicklist and ziplist: Memory‑efficient representations for lists and small collections.

Expiration and Eviction Policies

Redis combines periodic random sampling (default every 100 ms) with lazy deletion on key access. When memory usage exceeds maxmemory, one of the configurable eviction policies in redis.conf is applied:

maxmemory-policy volatile-lru

volatile-lru

: LRU among keys with an expiration. volatile-ttl: Evict keys that are closest to expiration. volatile-random: Randomly evict an expiring key. allkeys-lru: LRU across the entire keyspace. allkeys-random: Random eviction across all keys. noeviction: Reject writes once the limit is reached.

Atomicity and Thread Model

Redis processes one command at a time in a single thread, guaranteeing that each command is atomic without explicit locks. Newer versions can spawn optional I/O threads for network handling, but command execution remains single‑threaded.

The event loop monitors socket readiness (read/write/close), queues events, and processes them sequentially, ensuring deterministic execution order.

Transactions

Redis implements transactions with four primitives: MULTI: Starts a transaction; subsequent commands are queued. EXEC: Executes all queued commands atomically. DISCARD: Clears the queue and aborts the transaction. WATCH: Monitors one or more keys for changes; if any watched key is modified before EXEC, the transaction aborts.

Redis does not provide automatic rollback on runtime errors; commands that were successfully queued before the error remain applied.

Distributed Lock with SETNX

To create a lock, execute:

SETNX lock_key unique_id
EXPIRE lock_key 30   # optional expiration to avoid deadlock

Release the lock with DEL lock_key. For stronger safety, combine SETNX with GETSET and timestamps to ensure that only the lock owner can release it.

Cluster Solutions

Twemproxy : A proxy that performs consistent hashing across Redis nodes. It does not automatically rebalance when nodes are added or removed.

Codis : Similar to Twemproxy but adds data migration support for node changes.

Redis Cluster (3.0+) : Native sharding using 16,384 hash slots, built‑in replica support, and automatic failover.

Multi‑Machine Deployment and Consistency

Typical master‑slave replication provides read‑write separation. The master handles writes and propagates changes to one or more slaves, which serve read‑only traffic. Replication is asynchronous by default; synchronous replication can be enabled with the WAIT command or by configuring replica‑sync‑side‑cars for stricter consistency.

Handling Massive Request Volumes

Although command execution is single‑threaded, Redis leverages I/O multiplexing (select/epoll/kqueue) to serve thousands of concurrent client connections efficiently.

Common Performance Issues and Remedies

Avoid heavy persistence (RDB/AOF) on the master; offload to a slave.

Enable AOF on a slave with appendfsync everysec for reliable backups.

Place master and slaves in the same LAN to minimize replication latency.

Do not add slaves to an already overloaded master; instead, build a linear replication chain (master → slave1 → slave2 …) for stability.

Thread Model Details

The event loop consists of:

Socket I/O multiplexing layer (select/epoll/kqueue).

File‑event dispatcher that maps readiness events to handlers.

Handlers that perform the actual read/write/accept/close operations.

Even if multiple sockets become ready simultaneously, events are queued and processed one by one, guaranteeing sequential command execution.

This concise overview covers Redis persistence, cache‑related pitfalls and mitigations, data structures, eviction policies, atomic command execution, transaction semantics, distributed locking, clustering options, replication consistency, and performance tuning.