The “Ghost” Distributed Lock Issue: 3‑Day Debugging of Lock Failure

In a high‑traffic payment system, the business logic adds points after a successful order payment. The flow is: payment callback → acquire distributed lock (by userId) → read user points → add points → update DB → release lock. Despite the lock, two nearly simultaneous payments for the same user resulted in points being added twice.

1. Incident Symptoms

Business scenario: after order payment, add points. The logic is simple, but on a certain day two payments for the same user triggered two point increments almost at the same time, even though the code contained a distributed lock.

2. Code Inspection

The distributed lock is implemented with Redisson:

// Pseudo‑code
@Transactional
public void addPoints(Long userId, Integer points) {
    RLock lock = redissonClient.getLock("user:points:" + userId);
    try {
        // Acquire lock, wait 3 s, auto‑release after 10 s
        if (lock.tryLock(3, 10, TimeUnit.SECONDS)) {
            // Query points
            User user = userRepository.findById(userId);
            int currentPoints = user.getPoints();
            // Compute new points
            int newPoints = currentPoints + points;
            // Update points
            userRepository.updatePoints(userId, newPoints);
            // Send point‑change event …
        }
    } finally {
        if (lock.isHeldByCurrentThread()) {
            lock.unlock();
        }
    }
}

3. Investigation Process

Step 1: Check Logs

Using tracing, the timeline of two requests was observed:

Request A: 10:00:00.000 – lock acquired successfully
Request A: 10:00:00.100 – query points
Request A: 10:00:00.200 – update points

Request B: 10:00:00.050 – lock acquisition failed (waiting)
Request B: 10:00:00.250 – lock acquired ← A should have released the lock
Request B: 10:00:00.350 – query points
Request B: 10:00:00.450 – update points

The timeline looks correct—A acquires the lock first, B waits, then B acquires after A releases. Yet the final result shows both updates happened, meaning the updates overlapped.

Step 2: Locate the Real Issue

The problem lies in the mismatch between transaction boundaries and lock order. Spring’s @Transactional and Redisson’s lock have subtle ordering differences. The actual execution order observed:

1. Enter method
2. Open transaction ← transaction starts first
3. Acquire distributed lock ← lock is taken inside the transaction
4. Business logic
5. Commit transaction ← transaction commits after the lock is released?
6. Release lock ← but the transaction commit may still be in progress

Thus the lock is released while the transaction is still pending. When B acquires the lock, it reads the data before A’s transaction has been committed, leading to duplicate point updates.

Step 3: Confirm Root Cause

The lock release timing is earlier than the transaction commit, so B sees stale data and overwrites it.

4. Solutions

Solution 1: Put the Lock Outside the Transaction (Recommended)

public void addPoints(Long userId, Integer points) {
    RLock lock = redissonClient.getLock("user:points:" + userId);
    try {
        if (lock.tryLock(3, 10, TimeUnit.SECONDS)) {
            // ✅ Lock is outside the transaction, ensuring the lock is released only after the transaction commits
            doAddPoints(userId, points);
        }
    } finally {
        if (lock.isHeldByCurrentThread()) {
            lock.unlock();
        }
    }
}

@Transactional
public void doAddPoints(Long userId, Integer points) {
    // Business logic inside the transaction
    // Transaction commits, then the lock is already released
}

Execution order becomes:

Acquire lock → Open transaction → Business logic → Commit transaction → Release lock ✅

Solution 2: Adjust Transaction Isolation Level

@Transactional(isolation = Isolation.REPEATABLE_READ)
public void addPoints(Long userId, Integer points) {
    // ...
}

This can mitigate read‑write anomalies but does not address the fundamental lock‑transaction boundary mismatch.

Solution 3: Use Database Pessimistic Lock

@Transactional
public void addPoints(Long userId, Integer points) {
    // Replace Redis lock with a row‑level lock
    User user = userRepository.findByIdForUpdate(userId); // SELECT ... FOR UPDATE
    user.setPoints(user.getPoints() + points);
    userRepository.save(user);
}

Row‑level locks are naturally bound to the transaction, eliminating the boundary issue.

5. Common Distributed‑Lock Pitfalls Summary

Lock and transaction boundaries mismatch – symptom: lock released before transaction commits; solution: place lock outside the transaction.

Lock expiration too short – symptom: business not finished before lock auto‑releases; solution: set a reasonable timeout or use a watchdog.

Re‑entrant lock problems – symptom: same thread repeatedly acquires the lock; solution: use Redisson’s re‑entrant lock.

Incorrect lock release – symptom: thread A releases thread B’s lock; solution: check ownership before unlocking.

Master‑slave switch lock loss – symptom: lock disappears after Redis failover; solution: use RedLock or Zookeeper.

GC causing lock expiration – symptom: Full GC pauses the business, lock auto‑releases; solution: employ a watchdog to renew the lock.

6. Final Note

Lock release must occur after transaction commit.

This principle looks simple but is easily overlooked, especially when using AOP @Transactional.

If you also combine distributed locks with transactions, double‑check the order of lock acquisition/release relative to transaction boundaries.

# Quick troubleshooting commands
# 1. View Redis lock status
redis-cli KEYS "user:points:*"
redis-cli TTL "user:points:123"

# 2. View Redis slow log (investigate lock latency)
redis-cli SLOWLOG GET 10

# 3. View MySQL transaction status
SELECT * FROM information_schema.innodb_trx;