Redis Distributed Lock Failure Analysis and Safer Lock Implementation for High‑Concurrency Seckill

Incident Overview

The team used a Redis distributed lock for a flash‑sale (seckill) of 100 bottles of a scarce product, but the lock failure led to an oversell of 100 bottles, triggering a P0 incident and performance penalties.

Root Causes

1. No system‑level fault tolerance : User‑service overload caused gateway latency, allowing requests to exceed the lock’s 10‑second TTL.

2. Unsafe distributed lock : The lock was set with a fixed expiration; if a thread held the lock longer than the TTL, another thread could acquire it, and the original thread’s later unlock would delete the new lock.

3. Non‑atomic stock check : Stock verification used a get‑then‑compare pattern, which is not atomic under high concurrency, leading to oversell.

Initial Code Example

public SeckillActivityRequestVO seckillHandle(SeckillActivityRequestVO request) {
    SeckillActivityRequestVO response;
    String key = "key:" + request.getSeckillId;
    try {
        Boolean lockFlag = redisTemplate.opsForValue().setIfAbsent(key, "val", 10, TimeUnit.SECONDS);
        if (lockFlag) {
            // user validation, activity validation
            Object stock = redisTemplate.opsForHash().get(key+":info", "stock");
            assert stock != null;
            if (Integer.parseInt(stock.toString()) <= 0) {
                // business exception
            } else {
                redisTemplate.opsForHash().increment(key+":info", "stock", -1);
                // create order, emit event, build response
            }
        }
    } finally {
        // release lock
        stringRedisTemplate.delete("key");
    }
    return response;
}

Proposed Solutions

Safer Distributed Lock

Implement lock release only when the stored value matches, using a Lua script for atomic get‑and‑compare:

public void safedUnLock(String key, String val) {
    String luaScript = "local in = ARGV[1] local curr=redis.call('get', KEYS[1]) if in==curr then redis.call('del', KEYS[1]) end return 'OK'";
    RedisScript<String> redisScript = RedisScript.of(luaScript);
    redisTemplate.execute(redisScript, Collections.singletonList(key), Collections.singleton(val));
}

Atomic Stock Validation

Leverage Redis' atomic increment operation to decrement stock safely:

// redis returns the value after decrement, which is atomic
Long currStock = redisTemplate.opsForHash().increment("key", "stock", -1);

Refactored Seckill Handler

public SeckillActivityRequestVO seckillHandle(SeckillActivityRequestVO request) {
    SeckillActivityRequestVO response;
    String key = "key:" + request.getSeckillId();
    String val = UUID.randomUUID().toString();
    try {
        Boolean lockFlag = distributedLocker.lock(key, val, 10, TimeUnit.SECONDS);
        if (!lockFlag) {
            // business exception
        }
        // user and activity validation
        Long currStock = stringRedisTemplate.opsForHash().increment(key+":info", "stock", -1);
        if (currStock < 0) {
            // out of stock handling
        } else {
            // create order, emit event, build response
        }
    } finally {
        distributedLocker.safedUnLock(key, val);
    }
    return response;
}

Further Considerations

Even without a lock, using Redis' atomic decrement can prevent oversell, but a lock helps throttle traffic to downstream services. Alternatives like RedLock offer higher reliability at the cost of performance; the choice depends on the required reliability level.

Additional optimizations include sharding stock across servers and routing requests via consistent hashing to reduce contention.

Conclusion

Overselling of scarce items is a severe incident; the analysis shows that relying solely on a naive distributed lock is unsafe. By employing Lua‑based lock release and atomic stock operations, the system becomes more robust while still benefiting from traffic throttling when needed.