Building a Custom Redis Distributed Session: Token Design, Dual-Index Storage & Atomic Login

Introduction

The author illustrates the classic problem: after scaling a backend from one server to two behind Nginx, users were randomly logged out because requests hit different instances, each with its own in-memory HttpSession map. The root cause is that session state is bound to a single process's heap, conflicting with the load balancer's request distribution.

Concept Definitions

2.1 What is Session

Definition : Session is a server-side mechanism to maintain user state across requests. The server assigns a unique identifier (typically JSESSIONID) and stores associated state in memory.

Plain explanation : HTTP is stateless. Session gives the client a numbered ticket (sessionid); the server keeps a notebook (memory) keyed by that number.

2.2 What is Distributed Session

Definition : Moving session data from single-machine memory to a shared store (e.g., Redis) so multiple app instances can access the same session data, enabling horizontal scaling.

Plain explanation : Instead of each server keeping its private notebook, they all share a public locker (Redis).

2.3 Session vs Token

They are not opposing concepts but describe session management from different angles:

Perspective : Session = server-side state keeping; Token = client-held credential.

State location : Session = server memory / shared storage; Token = client (Cookie/Header).

Validation : Session = lookup by id on server; Token = self-contained or server lookup.

Typical implementations : Session = Servlet Session, Redis Session; Token = JWT, Opaque Token.

Key insight : In this custom Redis distributed session design, "Token" is simply the sessionid — a random string whose real state (user id, permissions, expiry) lives in Redis. This is the opposite of JWT's self-contained approach.

2.4 Three Core Concepts

Token (credential) : Random string held by client to identify session. Analogy: locker ticket with random number.

Session data : Session state in Redis (uid, permissions, device, IP). Analogy: items stored in the locker.

TTL (expiry) : Redis key remaining TTL, auto-evicted on expiry. Analogy: ticket validity period.

2.5 Authentication Session vs Resource Access Session

Authentication session : Login state established after login. Example: stay logged in for 7 days.

Resource access session : Per-request context. Example: current request's user id, permissions, trace id.

The article focuses on authentication sessions. Resource access sessions typically use ThreadLocal within a request.

Why Single-Machine Session Fails

3.1 Single-Machine Session Workflow

User first request → Tomcat creates HttpSession, generates JSESSIONID.

Response sets Set-Cookie: JSESSIONID=xxx.

Subsequent requests send cookie; Tomcat finds session by id.

Default 30-minute idle expiry.

Under the hood, Tomcat's StandardManager (or PersistentManager) maintains a ConcurrentHashMap<String, Session>. Request.getSession() retrieves by JSESSIONID. This map is process-local — a different machine has an empty map.

3.2 Breakage Under Multiple Instances

With a load balancer (Nginx/gateway) distributing requests across instances, a user's two requests may land on different servers. The second server's memory lacks the sessionid, so the user is treated as unauthenticated.

Root cause : Session state is pinned to one machine's process memory, conflicting with load balancer's "indiscriminate forwarding".

3.3 Two Solution Paths

Sticky Session : Load balancer hashes sessionid to pin user to same instance. Pros: simple. Cons: machine restart loses all its sessions, cannot scale smoothly, violates stateless ideal.

Shared Storage : Move session to Redis or other external store, all instances share. Pros: any instance handles any request, true horizontal scaling. Cons: adds one network hop and storage dependency.

Why sticky session is a band-aid :

Machine down = sessions lost : All users pinned to a restarted/crashed machine get logged out.

Uneven scaling : ip_hash distribution is fixed; new instances get no old traffic.

Rolling deploy blocked : Restarting instances kicks their users; no zero-downtime deploy.

Traffic imbalance : Hot IP segments overload one machine while others idle.

Therefore production almost always chooses shared storage, with Redis being the most common.

3.4 Why Redis

Session access pattern: high-frequency reads, low-frequency writes, low latency required, brief loss acceptable . Redis fits because:

Sub-millisecond access , negligible auth latency.

Rich expiry mechanisms ( EXPIRE / PERSIST), native TTL support.

Versatile data structures (String/Hash/Set) for organizing session data.

Mature HA (replication, Sentinel, Cluster) avoids single point of failure.

Can MySQL store sessions? Technically yes, but per-request DB queries are too costly; peak load would make DB the auth bottleneck. MySQL suits "persistent fallback", not high-frequency session storage.

Comparison with other stores:

Redis : ✅ First choice — in-memory, TTL, rich structures, mature HA.

Memcached : ⚠️ Usable — in-memory but no persistence, no rich structures, no optimized TTL eviction.

Database : ❌ Not recommended — disk I/O too slow for per-request auth.

Local cache (Caffeine) : ⚠️ Only as L1 cache — not cross-instance; can complement Redis as L2.

Analogy: Self-Service Locker

Map the system to a water park:

Customer → User

Front desk → Login endpoint

Locker → Redis

Ticket (random number) → Token

Items in locker (key, wristband, deposit record) → Session data

Ticket expiry → TTL

Multiple front desks → Multiple app instances

Key design points mapped:

Ticket number must be random — predictable numbers allow impersonation.

Locker has expiry — unclaimed items cleared after 30 days.

One customer can hold multiple tickets — phone and laptop sessions coexist.

Front desk can invalidate a ticket — suspicious login → clear locker, ticket instantly void.

Anti-Patterns and Correct Pattern

5.1 Anti-Pattern: Raw Servlet Session on Multi-Node

Using Nginx ip_hash (sticky session). Worked day one; next day rolling JVM restart caused all users on the restarted node to lose sessions. New third node got almost no traffic because ip_hash distribution is static. Problem : Sticky session turns "horizontal scaling" into "horizontal stacking" — machine count grows but single-machine failure blast radius stays the same.

5.2 Anti-Pattern: MD5(userId) as Token

// Anti-pattern
String token = DigestUtils.md5Hex(String.valueOf(userId));

Fatal flaws:

Predictable token — userId is auto-increment integer; attacker computes own token then enumerates nearby ids to hijack sessions.

Same user always same token — cannot support "multi-device login" or "kick old device"; new device login doesn't change token, old device stays valid.

Cannot actively invalidate — token derived from userId; only changing userId invalidates it.

5.3 Anti-Pattern: Write Token Then Write UID Set (Non-Atomic)

// Anti-pattern: two writes, crash in between leaves dirty data
redisson.getBucket("login:token:" + token).set(session, TTL);
// crash here → token exists but uid set not updated
redisson.getSet("login:uid:" + uid).add(token);

Non-atomic two-step write causes token to work but uid set missing it → kick-out misses tokens, multi-device management inaccurate. Correct approach uses Lua script for atomicity (see 7.3).

5.4 Correct Pattern: Random Token + Redis Dual-Index + Lua Atomic Write

Token generated with cryptographically secure random, session data in Redis, dual indexes (token→data and uid→token list), Lua script guarantees atomic write of both indexes. This design enables active kick-out, multi-device coexistence, sliding expiry, and atomic consistency.

Core 1: Token Generation Strategy Deep Dive

6.1 Token Requirements

Unpredictable : Token equals password; predictability enables enumeration attacks.

Globally unique : Collision causes session hijacking.

Moderate length : Too short → collisions; too long → bandwidth/storage waste.

6.2 Approach Comparison

MD5(userId)

: Predictable, insecure → ❌ UUID.randomUUID(): JDK uses SecureRandom, 128-bit → ✅ Usable, but contains - needing handling. SecureRandom + Hex: Cryptographically secure random source → ✅✅ Recommended.

Snowflake ID: Ordered but predictable, leaks scale → ❌ Unsuitable for tokens. Math.random() / Random: Linear congruential, predictable → ❌

6.3 Recommended Implementation: SecureRandom + Hex

import java.security.SecureRandom;

public class TokenGenerator {
    private static final SecureRandom SECURE_RANDOM = new SecureRandom();
    private static final char[] HEX = "0123456789abcdef".toCharArray();

    /** Generate 32-char random token (128-bit entropy) */
    public static String generate() {
        byte[] bytes = new byte[16]; // 16 bytes = 128 bits
        SECURE_RANDOM.nextBytes(bytes);
        char[] chars = new char[bytes.length * 2];
        for (int i = 0; i < bytes.length; i++) {
            int v = bytes[i] & 0xFF;
            chars[i * 2] = HEX[v >>> 4];
            chars[i * 2 + 1] = HEX[v & 0x0F];
        }
        return new String(chars);
    }
}

Why not Math.random() or Random ? Random uses linear congruential generator; given seed, entire sequence is predictable. Math.random() wraps Random. Token is a session credential — predictability lets attackers enumerate tokens and impersonate any user. SecureRandom uses system entropy sources (e.g., /dev/urandom), cryptographically unpredictable, the correct choice for tokens.

6.4 Entropy and Brute-Force Analysis

128-bit entropy meaning : Even at 1 trillion attempts/sec (10^12), exhausting 2^128 space takes 2^128 / 10^12 ≈ 3.4 × 10^26 seconds ≈ 10^19 years — far exceeding universe age (~1.4 × 10^10 years). 32-char hex token is security-sufficient.

Versus 64-bit token : 2^64 ≈ 1.8 × 10^19; at 10^12 attempts/sec needs ~585 years. Seems enough, but distributed attacks over years give theoretical risk. Production practice starts at 128-bit.

Token length vs collision probability : 32-char hex (128-bit), by birthday paradox, ~2^64 ≈ 1.8 × 10^19 tokens for 50% collision chance — no system reaches this scale, so global uniqueness is assured.

Side note : UUID v4 also has 122 random bits, comparable security. But UUID contains - (e.g., 550e8400-e29b-... ); as token either strip - or use Hex directly. Both work; SecureRandom + Hex is more compact and controllable.

Core 2: Redis Storage Structure Design

This is the most critical design decision.

7.1 Key Organization

login:token:{token}

(Hash): uid, device, loginIp, loginTime, permission snapshot, absoluteExpireAt. TTL: session expiry (e.g., 7 days). login:uid:{uid} (Set): all valid tokens for that user. TTL: none (maintained by cleanup logic).

7.2 Why Hash Instead of String(JSON)

Partial update : HSET single field, efficient vs read entire JSON, deserialize, modify, serialize, write back.

Single-field query : HGET saves bandwidth vs GET fetches whole JSON.

Atomic read-modify-write : Hash not atomic across fields; String(JSON) can do whole-object CAS.

Memory usage : Few fields → ziplist compression, saves space vs JSON string stored verbatim.

Session scenario has fixed fields (5-6), with partial update needs (e.g., renew absoluteExpireAt on sliding expiry, update IP on anomaly detection). Hash fits better.

7.3 Lua Script Guarantees Atomic Write

Login must write both token→session and uid→token set atomically — otherwise crash leaves dirty data. Lua script:

private static final String LOGIN_LUA = """
    redis.call('HSET', KEYS[1], 'uid', ARGV[1], 'device', ARGV[2],
                 'loginIp', ARGV[3], 'loginTime', ARGV[4],
                 'perms', ARGV[5], 'absoluteExpireAt', ARGV[6])
    redis.call('EXPIRE', KEYS[1], ARGV[7])
    redis.call('SADD', KEYS[2], ARGV[8])
    return 1
    """;

public String login(Long uid, String device, String loginIp, Set<String> permissions) {
    String token = TokenGenerator.generate();
    String tokenKey = TOKEN_PREFIX + token;
    String uidKey = UID_PREFIX + uid;
    long ttlSeconds = SESSION_TTL.toSeconds();
    long absoluteExpireAt = Instant.now().plus(ABSOLUTE_TTL).getEpochSecond();

    RScript script = redisson.getScript();
    script.eval(RScript.Mode.READ_WRITE, LOGIN_LUA, RScript.ReturnType.INTEGER,
            List.of(tokenKey, uidKey),
            String.valueOf(uid), device, loginIp,
            String.valueOf(Instant.now().getEpochSecond()),
            String.join(",", permissions),
            String.valueOf(absoluteExpireAt),
            String.valueOf(ttlSeconds),
            token);
    return token;
}

Lua atomicity principle : Redis executes commands single-threaded; a Lua script runs uninterrupted — all commands in script either fully execute or none do, no other client commands interleave. This is key to keeping the two indexes consistent.

7.4 Whether to Store Permission Snapshot in Session

Store permission snapshot : Auth check avoids DB, fast. Con: permission changes not immediate.

Don't store, query DB each request : Permissions real-time. Con: per-request DB hit, high load.

Store snapshot + refresh on change : Fast and near-real-time. Con: complex implementation.

Compromise : Store permissions in session for "hot path", but on permission change actively refresh all that user's sessions (iterate uid→token set, HSET update perms field).

7.5 Why uid → token Set Reverse Index

Enable "kick user offline": know uid → find all tokens → delete each.

Enforce "max login devices": check set size before login, evict oldest if over limit.

Provide "user's session list": show "currently logged in on these devices".

Permission change refresh: iterate set, batch update.

Core 3: Complete Login Core Code

import org.redisson.api.*;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Set;

@Service
public class LoginService {

    public static final Duration SESSION_TTL = Duration.ofDays(7);   // sliding expiry 7 days
    public static final Duration ABSOLUTE_TTL = Duration.ofDays(30); // absolute expiry 30 days
    private static final String TOKEN_PREFIX = "login:token:";
    private static final String UID_PREFIX = "login:uid:";
    private static final int MAX_DEVICES = 5;

    private static final String LOGIN_LUA = """
        redis.call('HSET', KEYS[1], 'uid', ARGV[1], 'device', ARGV[2],
                     'loginIp', ARGV[3], 'loginTime', ARGV[4],
                     'perms', ARGV[5], 'absoluteExpireAt', ARGV[6],
                     'token', ARGV[8])
        redis.call('EXPIRE', KEYS[1], ARGV[7])
        redis.call('SADD', KEYS[2], ARGV[8])
        return 1
        """;

    private final RedissonClient redisson;

    public LoginService(RedissonClient redisson) {
        this.redisson = redisson;
    }

    /** Login: verify credentials, create session, return Token */
    public String login(Long uid, String device, String loginIp, Set<String> permissions) {
        String token = TokenGenerator.generate();
        String tokenKey = TOKEN_PREFIX + token;
        String uidKey = UID_PREFIX + uid;

        long absoluteExpireAt = Instant.now().plus(ABSOLUTE_TTL).getEpochSecond();

        redisson.getScript().eval(RScript.Mode.READ_WRITE, LOGIN_LUA,
                RScript.ReturnType.INTEGER,
                List.of(tokenKey, uidKey),
                String.valueOf(uid), device, loginIp,
                String.valueOf(Instant.now().getEpochSecond()),
                String.join(",", permissions),
                String.valueOf(absoluteExpireAt),
                String.valueOf(SESSION_TTL.toSeconds()),
                token);

        // Enforce max devices, evict oldest
        evictIfExceed(uid, MAX_DEVICES);
        return token;
    }

    /** Evict oldest token when exceeding max devices */
    public void evictIfExceed(Long uid, int max) {
        RSet<String> uidSet = redisson.getSet(UID_PREFIX + uid);
        if (uidSet.size() <= max) return;

        List<SessionInfo> sessions = uidSet.readAll().stream()
                .map(t -> redisson.<SessionInfo>getBucket(TOKEN_PREFIX + t).get())
                .filter(Objects::nonNull)
                .sorted(Comparator.comparing(SessionInfo::getLoginTime))
                .toList();

        int toEvict = sessions.size() - max;
        for (int i = 0; i < toEvict; i++) {
            logout(uid, sessions.get(i).getToken());
        }
    }

    /** Logout: delete specific token's session */
    public void logout(Long uid, String token) {
        // Lua ensures two-step atomic
        redisson.getScript().eval(RScript.Mode.READ_WRITE,
                "redis.call('DEL', KEYS[1]) redis.call('SREM', KEYS[2], ARGV[1]) return 1",
                RScript.ReturnType.INTEGER,
                List.of(TOKEN_PREFIX + token, UID_PREFIX + uid),
                token);
    }

    /** Kick out: delete all sessions for a user */
    public void kickOut(Long uid) {
        RSet<String> uidSet = redisson.getSet(UID_PREFIX + uid);
        Set<String> tokens = uidSet.readAll();
        for (String token : tokens) {
            redisson.getBucket(TOKEN_PREFIX + token).delete();
        }
        uidSet.delete();
    }

    /** Refresh session (sliding renewal) */
    public boolean refresh(String token) {
        RBucket<SessionInfo> bucket = redisson.getBucket(TOKEN_PREFIX + token);
        SessionInfo session = bucket.get();
        if (session == null) return false;

        Instant now = Instant.now();
        if (now.isAfter(session.getAbsoluteExpireAt())) {
            bucket.delete(); // past absolute expiry, force invalidate
            return false;
        }
        Duration remaining = Duration.between(now, session.getAbsoluteExpireAt());
        Duration ttl = remaining.compareTo(SESSION_TTL) < 0 ? remaining : SESSION_TTL;
        bucket.expire(ttl);
        return true;
    }
}

To be continued…