Unlocking Redis: Deep Dive into Data Structures, Persistence, and High Availability

What Is NoSQL?

NoSQL stands for "not only SQL". Relational databases store data in fixed rows and columns, while non‑relational databases have no fixed format and can scale horizontally. NoSQL refers to these non‑relational stores, which have grown rapidly with the rise of Web 2.0 and big‑data workloads; Redis is the fastest‑growing NoSQL database.

Traditional RDBMS vs NoSQL

RDBMS
 - Structured schema
 - Fixed SQL
 - Data and relationships stored in tables (rows/columns)
 - DML/DDL commands
 - Strong ACID consistency
 - Basic transactions

NoSQL
 - Not limited to data only
 - No fixed query language
 - Key‑value (Redis), column (HBase), document (MongoDB), graph (Neo4j)
 - BASE consistency (basic availability, soft state, eventual consistency)

What Is Redis?

Redis = Remote Dictionary Server, an open‑source, ANSI‑C implementation that supports networking, can store data in memory with optional persistence, and provides APIs for many languages.

Like Memcached, Redis caches data in memory for speed, but it also periodically writes updates to disk and supports master‑slave synchronization.

Redis Five Basic Data Types

Redis is an in‑memory data‑structure server that can be used as a database, cache, or message‑queue broker. It supports strings, hashes, lists, sets, sorted sets, bitmaps, HyperLogLog, and geospatial indexes. The following sections focus on the five fundamental types and their underlying storage structures.

1. String

String is the most basic and frequently used type. Other types are built on top of it.

Maximum size 512 MB.

Can store plain text, JSON/XML, binary images, audio, or numeric strings.

Application Scenarios

Cache: Accelerates reads and reduces backend load.

Counter: Real‑time counting with optional persistence to a database.

Statistics: E.g., per‑user counters (uid → count).

Shared session: Centralized session storage for fast access.

2. List

Lists store ordered strings and can act as queues or stacks.

Pre‑Redis 3.2: Implemented with ziplist or doubly‑linked list.

Since Redis 3.2: Implemented with quicklist (a linked list of ziplist nodes).

Conditions for ziplist: element length < 64 bytes and total elements < 512.

Application Scenarios

Message queue: Use LPUSH / BRPOP for producer‑consumer patterns.

Pagination: Fast range queries for ordered article lists.

3. Set

Sets are unordered collections of unique elements.

Implemented with hashtable or intset.

Intset used when all elements are integers and count < 512.

Application Scenarios

Tagging: Group users by common interests.

Mutual friends: Find common connections.

Unique IP counting: Real‑time distinct visitor statistics.

4. Sorted Set (Zset)

Zsets keep elements unique and assign a score for ordering.

Encoding: ziplist for small sets, skiplist for larger sets.

Application Scenarios

Leaderboards: Rank items by score (e.g., video views).

Weighted queue: Prioritize tasks by score.

5. Hash

Hashes map fields to values, ideal for representing objects.

Small hashes use ziplist; larger ones use dict.

Application Scenarios

Store relational rows: field‑value mapping per record.

User profiles: Fast retrieval without DB round‑trips.

Underlying Storage Structure

Redis stores all data in a large hashmap where each dictEntry holds a key and a redisObject value.

The redisObject structure contains type, encoding, LRU, reference count, and a pointer to the actual data.

typedef struct redisObject {
    unsigned type:4;        // object type (string, list, hash, set, zset, ...)
    unsigned encoding:4;    // internal encoding
    unsigned lru:22;        // LRU clock
    int refcount;           // reference count
    void *ptr;              // pointer to actual value
} robj;

Eight possible encodings are used: INT, EMBSTR, RAW, HT, LINKEDLIST, ZIPLIST, INTSET, SKIPLIST.

String Storage (SDS)

Redis strings use the SDS (Simple Dynamic String) structure, which stores length, allocated size, flags, and the buffer. For strings ≤ 44 bytes, Redis uses the embstr encoding (single allocation). Larger strings use raw encoding.

struct __attribute__((__packed__)) sdshdr8 {
    uint8_t len;      // used length
    uint8_t alloc;    // allocated size (excluding header)
    unsigned char flags;
    char buf[];      // actual bytes
};

SDS provides O(1) length retrieval, prevents buffer overflow, and reduces reallocations.

List Storage

Before Redis 3.2, lists used ziplist (compact) or linkedlist (doubly linked). Since 3.2, Redis uses quicklist, a linked list of ziplist nodes, combining low memory usage with fast push/pop.

typedef struct quicklist {
    struct quicklistNode *head, *tail;
    unsigned long count;
    unsigned int len;
    int fill;          // max ziplist size (list-max-ziplist-size)
    unsigned int compress; // compression depth (list-compress-depth)
} quicklist;

Each quicklistNode holds a ziplist pointer, size, element count, encoding, and container flags.

Hash Storage

Small hashes use ziplist; larger ones use a dict (hash table). The dict consists of two hash tables for incremental rehashing.

typedef struct dictEntry {
    void *key;
    union { void *val; uint64_t u64; int64_t s64; } v;
    struct dictEntry *next;
} dictEntry;

typedef struct dict {
    dictType *type;
    void *privdata;
    dictht ht[2];
    long rehashidx; // -1 when not rehashing
    int iterators;
} dict;

Set Storage

Sets use intset for small integer collections or a dict for general data.

typedef struct intset {
    uint32_t encoding; // 16/32/64 bits per element
    uint32_t length;   // number of elements
    int8_t contents[]; // sorted, unique integers
} intset;

Zset Storage

Zsets are encoded as ziplist (compact) or skiplist (fast range queries). Skiplist nodes store a score, member object, forward pointers, spans, and a backward pointer.

typedef struct zskiplistNode {
    robj *obj;          // member
    double score;       // sorting score
    struct zskiplistNode *backward;
    struct zskiplistLevel {
        struct zskiplistNode *forward;
        unsigned int span;
    } level[];
} zskiplistNode;

Special Data Types

Geospatial

Geospatial data (latitude, longitude, name) are stored in a sorted set using Geohash as the score, enabling radius queries via GEORADIUS and GEORADIUSBYMEMBER.

HyperLogLog

HyperLogLog provides cardinality estimation using a fixed 12 KB structure, capable of approximating up to 2⁶⁴ distinct elements with a small error margin.

Bitmaps

Bitmaps treat a string as a bit array (max 512 MB). They allow setting, clearing, and testing individual bits, useful for tracking binary states such as user activity.

Redis Transactions

Redis transactions batch multiple commands to be executed atomically. A transaction consists of MULTI, command queuing, and EXEC. Syntax errors abort the whole transaction; runtime errors do not roll back, so Redis relies on the programmer to avoid logical mistakes.

Persistence

RDB (Snapshot)

RDB creates a compressed binary snapshot of the dataset. It can be triggered manually ( SAVE, BGSAVE) or automatically via the save configuration (e.g., save 900 1, save 300 10, save 60 10000).

During BGSAVE, Redis forks a child process that writes the snapshot to a temporary file; the parent continues serving clients. After completion, the temporary file replaces the old dump.rdb.

Advantages: low memory overhead, fast restart, compact file. Disadvantages: possible data loss between snapshots, fork overhead for large datasets.

AOF (Append‑Only File)

AOF logs every write command in the Redis protocol format. It can be configured with appendfsync policies ( always, everysec, no). Over time the AOF grows; BGREWRITEAOF rewrites it into a compact form by replaying the current dataset and writing a minimal set of commands.

Advantages: higher durability (especially with always), ability to recover exact command sequence. Disadvantages: larger files, slower restart.

RDB vs AOF Comparison

Item

RDB

AOF

Startup priority

Low

High

File size

Small

Large

Recovery speed

Fast

Slow

Data safety

Possible loss

Configurable (always/everysec)

Publish/Subscribe

Redis implements a message‑passing model where publishers send messages to channels and subscribers receive them. Subscriptions can be exact channel names or patterns using wildcards ( *, ?).

Data structures: pubsub_channels: a dict mapping channel names to a list of subscribed clients. pubsub_patterns: a list of pubsubPattern structs, each holding a client and a pattern.

When a message is published, Redis sends it to all clients subscribed to the exact channel and to all pattern‑subscribed clients whose pattern matches the channel.

Master‑Slave Replication

Replication provides data redundancy, fault recovery, load balancing, and forms the basis for Sentinel and cluster high availability.

Full replication consists of three phases:

Handshake: the slave sends PSYNC ? -1, the master replies with FULLRESYNC and its run‑ID and offset.

Data transfer: the master creates an RDB snapshot (via BGSAVE) and streams it to the slave, which loads it after clearing its own data.

Command propagation: the master buffers write commands received after the snapshot and streams them to the slave, which replays them to catch up.

Incremental replication then streams only new write commands.

Sentinel Mechanism

Sentinel provides automatic failover for Redis masters.

Monitoring: every second each Sentinel pings masters, slaves, and other Sentinels; every 2 seconds they exchange hello messages on the __sentinel__:hello channel; every 10 seconds they fetch INFO from data nodes.

Subjective down: a Sentinel marks a node down after down-after-milliseconds without a reply.

Objective down: when a quorum (majority) of Sentinels agree a master is down.

Leader election: Sentinels use a Raft‑like voting process via SENTINEL IS-MASTER-DOWN-BY-ADDR. The first to obtain a majority becomes the leader.

Failover: the leader selects a slave (based on offset, priority, run‑ID) and promotes it with SLAVEOF NO ONE. Remaining slaves are reconfigured to replicate the new master.

Cache Issues

Cache Penetration

Occurs when requests target keys that do not exist in both cache and database, causing repeated DB hits. Solutions:

Validate request parameters (e.g., reject negative IDs).

Cache null results with a short TTL (e.g., 30 seconds).

Use a Bloom filter to pre‑check key existence.

Cache Breakdown (Cache Stampede)

Happens when a hot key expires and many concurrent requests miss the cache, overwhelming the DB. Solutions:

Never let hot data expire (set infinite TTL).

Rate‑limit or circuit‑break critical endpoints.

Use a mutex (e.g., distributed lock) so only one request rebuilds the cache.

Cache Avalanche

Massive simultaneous expiration of many keys leads to a sudden DB surge. Solutions:

Assign random TTLs to spread expirations.

Distribute hot data across multiple cache nodes.

Keep truly hot keys from expiring.

Conclusion

Each technology discussed forms a large ecosystem; continuous learning and practice are essential for mastering Redis and building robust, high‑performance systems.