Why Redis Distributed Locks Can Fail: Real-World Over‑Sale Case & Fixes

Preface

Using Redis for distributed locks is no longer new. This article shares a real incident from a flash‑sale of a scarce product that resulted in severe overselling, and presents the analysis and solutions.

Incident Scene

The flash‑sale of 100 bottles of a rare liquor was oversold, triggering a P0‑level incident and performance penalties for the whole team.

Root Causes

Massive user registration for the event put the user‑service under high load, causing request latency that exceeded the 10‑second lock timeout. When the lock expired, subsequent requests acquired the lock, and later the original request released it, unintentionally unlocking others and allowing duplicate purchases. Additionally, the stock check was non‑atomic.

Analysis

Three main safety issues were identified:

No fallback handling for overloaded user services.

Distributed lock was not truly safe. If a thread holds the lock longer than its TTL, another thread can acquire it, and when the first thread finally releases the lock it may delete the second thread's lock.

Non‑atomic stock verification. The get‑and‑compare approach leads to inaccurate stock counts under concurrency.

Solutions

Implement a Safer Distributed Lock

Use a unique value for the lock key and release the lock only when the stored value matches, implemented via a Lua script:

public void safedUnLock(String key, String val) {
    String luaScript = "local in = ARGV[1] local curr=redis.call('get', KEYS[1]) if in==curr then redis.call('del', KEYS[1]) end return 'OK'";
    RedisScript<String> redisScript = RedisScript.of(luaScript);
    redisTemplate.execute(redisScript, Collections.singletonList(key), Collections.singleton(val));
}

Make Stock Check Atomic

Leverage Redis's atomic increment operation for stock deduction:

// redis returns the result after the operation atomically
Long currStock = redisTemplate.opsForHash().increment("key", "stock", -1);

Refactored Business Logic

Introduce a DistributedLocker class and use the safer lock and atomic stock update:

public SeckillActivityRequestVO seckillHandle(SeckillActivityRequestVO request) {
    SeckillActivityRequestVO response;
    String key = "key:" + request.getSeckillId();
    String val = UUID.randomUUID().toString();
    try {
        Boolean lockFlag = distributedLocker.lock(key, val, 10, TimeUnit.SECONDS);
        if (!lockFlag) {
            // handle business exception
        }
        // user validation omitted for brevity
        Long currStock = stringRedisTemplate.opsForHash().increment(key + ":info", "stock", -1);
        if (currStock < 0) {
            log.error("[Seckill] No stock");
            // handle out‑of‑stock
        } else {
            // generate order, publish event, build response
        }
    } finally {
        distributedLocker.safedUnLock(key, val);
    }
    return response;
}

Deep Thinking

Is a Distributed Lock Necessary?

While Redis's atomic decrement can prevent overselling, the lock protects other services from excessive load during the flash‑sale.

Lock Selection

RedLock offers higher reliability at the cost of performance; for this scenario, a simple lock with proper TTL and safe release is more cost‑effective.

Further Optimizations

By sharding stock across cluster nodes and routing requests based on user ID hash, stock can be managed in‑process with a ConcurrentHashMap, eliminating the need for Redis entirely, though dynamic scaling would then require additional handling.

Conclusion

Overselling scarce goods can cause severe business impact. The incident taught that even seemingly safe code can become a fatal flaw under high concurrency, emphasizing the need for thorough design, continuous learning, and proper use of atomic operations and distributed locks.