Why Our Flash Sale Oversold: Lessons from a Distributed Lock Failure

Background

Our project uses a distributed lock for flash‑sale orders. During a high‑profile "Flying Moutai" sale with 100 bottles, the stock was oversold, triggering a P0‑level incident and performance penalties for the whole team.

Incident Scene

Although the sale interface had never caused problems before, the unprecedented demand caused massive spikes in user‑validation requests, overwhelming the user service and causing gateway response delays beyond 10 seconds (the lock timeout).

Root Cause

The lock expired while a request was still processing, allowing another thread to acquire the same lock. When the original thread finally released the lock, it unintentionally released the new thread’s lock, creating a race condition that let multiple requests decrement the same stock.

Additionally, the stock check was non‑atomic (a get‑and‑compare pattern), so concurrent requests could read stale stock values and still succeed, leading to overselling.

Solution

We introduced a safer distributed‑lock implementation that ties the lock value to a unique UUID and releases the lock only if the stored value matches, using a Lua script for atomicity:

public void safedUnLock(String key, String val) {
    String luaScript = "local in = ARGV[1] local curr=redis.call('get', KEYS[1]) if in==curr then redis.call('del', KEYS[1]) end return 'OK'";
    RedisScript<String> redisScript = RedisScript.of(luaScript);
    redisTemplate.execute(redisScript, Collections.singletonList(key), Collections.singleton(val));
}

We also simplified inventory checks by relying on Redis’s atomic increment operation:

Long currStock = stringRedisTemplate.opsForHash().increment(key+":info", "stock", -1);
if (currStock < 0) { /* out of stock */ }
else { /* generate order */ }

Finally, we refactored the business logic into a dedicated DistributedLocker class, using the UUID‑based lock and atomic stock decrement, and ensured the lock is always released in a finally block.

Deep Thinking

Even with a safer lock, eliminating the lock entirely and using Redis’s atomic decrement can prevent overselling, but it increases load on downstream services. A balanced approach—using a lightweight lock for traffic shaping while leveraging Redis atomicity for stock updates—offers the best trade‑off.

Conclusion

The incident highlighted the danger of assuming distributed locks are infallible. Proper lock ownership verification and atomic inventory operations are essential to avoid overselling in high‑concurrency flash‑sale scenarios.