Why Does Redis Store the Same Data Twice?

Redis uses two different internal representations for its collection types to optimize both memory usage and operation performance. For ordinary sets, the underlying structures are intset (an integer set) and a hash table; for sorted sets, they are a skiplist and a ziplist.

Set object encoding

The set object can be encoded as either intset or hashtable. intset stores only integer values using one of three integer types ( int16_t, int32_t, int64_t) determined by the encoding field. The source definition ( intset.h) is:

typedef struct intset {
    uint32_t encoding;   // encoding type
    uint32_t length;     // number of elements
    int8_t contents[];   // actual elements
} intset;

When the set contains only integers and the element count does not exceed set-max-intset-entries (default 512), Redis chooses the intset encoding.

Integer set upgrade process

If a new element requires a larger integer type, Redis upgrades the intset in four steps:

Expand the underlying array based on the new element’s size.

Convert existing elements to the larger type and copy them backward.

Insert the new element at the head or tail, depending on its value.

Update the encoding and length fields.

Upgrading is irreversible; once an intset is upgraded, it never downgrades.

Upgrade example

Assume a set encoded as int16_t with three elements. Inserting the integer 50000 (requires int32_t) triggers an upgrade:

Allocate new space: 4 * 32 - 48 = 80 bytes.

Move existing elements to the new array positions (64‑95, 32‑63, 0‑31).

Place 50000 at positions 96‑127.

Update encoding to INTSET_ENC_INT32 and adjust length.

Hashtable encoding

If the set contains any non‑integer element or exceeds the entry limit, Redis stores it as a hash table. The article references the earlier hash object analysis for details.

Common set commands

sadd key member1 member2

– add members. sismember key member – test membership. srem key member1 member2 – remove members. smove src dst member – move a member. smembers key – list all members.

Sorted‑set object encoding

Sorted sets store each element with a double‑precision score. The two possible encodings are skiplist and ziplist.

Skiplist encoding

A skiplist (jump list) provides O(log N) range queries while maintaining order. Each node ( zskiplistNode, defined in server.h) contains:

typedef struct zskiplistNode {
    sds ele;               // element
    double score;          // score
    struct zskiplistNode *backward;
    struct zskiplistLevel {
        struct zskiplistNode *forward; // forward pointer
        unsigned long span;             // distance to next node
    } level[];
} zskiplistNode;

The skiplist is wrapped by a zset object, which also holds a hash table for O(1) element lookup:

typedef struct zset {
    dict *dict;      // hash table (member → score)
    zskiplist *zsl; // skiplist for ordered traversal
} zset;

This dual structure allows constant‑time lookups via the hash table and logarithmic‑time range queries via the skiplist.

Why both dictionary and skiplist?

Using only a skiplist would give O(log N) lookups; using only a hash table would give O(1) lookups but no ordering. Combining them gives the best of both worlds, which is a key design insight of Redis.

Ziplist encoding

When a sorted set has fewer than 128 elements and the total length of all elements is under 64 bytes (configurable via zset-max-ziplist-entries and zset-max-ziplist-value), Redis stores it as a compact ziplist.

Common sorted‑set commands

zadd key score1 member1 ...

– add members with scores. zscore key member – get a member’s score. zincrby key increment member – increment a score. zrange key start stop – get members in score order. zrevrange key start stop – get members in reverse order. zrangebyscore key min max – range query by score. zlexcount key min max – count members by lexical range.

Conclusion

The article analyses the internal storage mechanisms of Redis sets and sorted sets, explains the intset and skiplist implementations, demonstrates how integer sets upgrade their encoding, and clarifies why Redis stores ordered sets using both a dictionary and a skiplist to achieve O(1) lookups together with efficient ordered range operations without redundant data storage.