Spring Boot Distributed Locks: From Redisson Watchdog to RedLock Controversy

Two Classic Incidents

Incident 1: During a flash sale, 103 items sold for 100 inventory. Six pods concurrently executed "check inventory → judge → deduct" without cluster-wide locking; single-JVM synchronized only protects one process.

Incident 2: Thread A holds lock for 15 seconds, lock expires at 10 seconds, thread B acquires lock, thread A finishes and deletes B's lock, thread C enters — three threads simultaneously hold the same lock.

These incidents define four hard requirements: mutual exclusion, reentrancy, deadlock prevention (auto-expiry on holder crash), and fault tolerance (lock service failure must not halt business).

Three Implementations & Selection

Database lock (pessimistic/unique key): INSERT unique key or SELECT FOR UPDATE. Zero new components, strong consistency; but poor performance, no auto-expiry, row-lock limits. Suits low-frequency tasks (e.g., prevent duplicate cron runs).

Redis lock: SET NX EX + TTL. Highest performance, mature ecosystem (Redisson); trade-off: async master-replica replication creates "lock loss" window, TTL vs business duration hard to align. Mainstream choice for internet companies.

ZooKeeper / etcd lock: Ephemeral sequential nodes + watch predecessor. CP semantics, natural deadlock prevention (session expiry deletes node); cost: performance an order of magnitude lower, higher ops burden. Suits strong-consistency, low-frequency scenarios (leader election, metadata).

Selection mnemonic: High-frequency mutex → Redis (accept tiny failure probability + business idempotency fallback); strong-consistency red line → ZK/etcd; small volume, want simplicity → database. RedLock is not a silver bullet (see section 5).

Core Flow: From SET NX to Watchdog

3.1 Common Wrong Patterns

// Wrong 1: setnx and expire two steps
redis.setnx(key, "1");
redis.expire(key, 10); // crash in between = deadlock

// Wrong 2: fixed value, no holder verification on release
redis.set(key, "1", 10, SECONDS);
// ...business times out, lock expires
redis.delete(key); // deletes someone else's lock

// Wrong 3: get then del, non-atomic
if (token.equals(redis.get(key))) {
    redis.delete(key); // lock may change hands between get and del
}

3.2 Correct Native Implementation (Interview Essential)

String token = UUID.randomUUID().toString(); // unique holder identifier

// Atomic lock: SET key value NX EX 10
Boolean ok = redis.opsForValue()
    .setIfAbsent(key, token, Duration.ofSeconds(10));
if (!Boolean.TRUE.equals(ok)) {
    throw new BizException("Failed to acquire lock");
}
try {
    // business... may exceed 10s?
    // This is what watchdog solves
} finally {
    // Lua: delete only if value matches, atomic release
    String lua = "if redis.call('get',KEYS[1])==ARGV[1] "
        + "then return redis.call('del',KEYS[1]) "
        + "else return 0 end";
    redis.execute(new DefaultRedisScript<>(lua, Long.class),
        List.of(key), token);
}

Three pillars: atomic locking ( setIfAbsent single command with TTL), unique token (identifies holder, prevents accidental deletion), Lua atomic release (check-and-delete inseparable). Remaining hard problem: business runs 15s, lock expires at 10s. Increase lease to 60s? If holder crashes at 5s, others wait 55s. This is exactly where Redisson watchdog shines.

3.3 Redisson Usage Pattern

RLock lock = redisson.getLock("order:" + orderId);

// Key: omit leaseTime, only pass waitTime
// This activates watchdog auto-renewal!
boolean locked = lock.tryLock(3, TimeUnit.SECONDS);
if (!locked) {
    throw new BizException("Lock contention failed");
}
try {
    // check inventory → deduct → place order
} finally {
    // Prevent IllegalMonitorStateException:
    // lock already expired and transferred, direct unlock throws
    if (lock.isHeldByCurrentThread()) {
        lock.unlock();
    }
}

Core Source Code: Redisson's Three Lua Scripts

RedissonLock's lock, renew, unlock are all Lua scripts guaranteeing atomic multi-command execution. Lock data structure is a Hash : field = "clientId + threadId", value = reentry count.

4.1 Lock Acquisition tryLockInnerAsync (Simplified)

// KEYS[1]=lock name  ARGV[1]=lease time
// ARGV[2]=thread identifier (for reentry)
-- Case 1: lock doesn't exist, acquire directly
if (redis.call('exists', KEYS[1]) == 0) then
    redis.call('hincrby', KEYS[1], ARGV[2], 1);
    redis.call('pexpire', KEYS[1], ARGV[1]);
    return nil;
end;
-- Case 2: lock exists and is mine → reentry +1
if (redis.call('hexists', KEYS[1], ARGV[2]) == 1) then
    redis.call('hincrby', KEYS[1], ARGV[2], 1);
    redis.call('pexpire', KEYS[1], ARGV[1]);
    return nil;
end;
-- Case 3: someone else's lock → return remaining TTL
return redis.call('pttl', KEYS[1]);

Three return values reveal reentrant design: nil = lock acquired (first time or reentry), pttl returns milliseconds = failure — client gets remaining time, uses Netty scheduled task + pub/sub (broadcast unlock on channel) to implement "wake up instead of polling", the foundation of efficient tryLock(waitTime) waiting.

4.2 Watchdog: renewExpiration Renewal Mechanism

// Default lease 30s (lockWatchdogTimeout)
// After successful lock, schedule a task:
// Every 30/3 = 10s check and renew

private void renewExpiration() {
    ExpirationEntry ee = EXPIRATION_RENEWAL_MAP.get(getEntryName());
    Timeout task = commandExecutor.getConnectionManager()
        .newTimeout(t -> {
            // Async pexpire renew to 30s
            renewExpirationAsync(threadId);
            // Renewal success → schedule next
            // 10s periodic task
        }, internalLockLeaseTime / 3, TimeUnit.MILLISECONDS);
    ee.setTimeout(task);
}

Three watchdog facts: ① Only starts when leaseTime not specified (source: leaseTime == -1 triggers watchdog branch); ② Renewal interval = lease/3 (30s lease → renew every 10s); ③ Client crash kills scheduled task, lock auto-releases within 30s — deadlock prevention vs business duration contradiction resolved .

4.3 Unlock (Reentrant Decrement)

// KEYS[1]=lock name KEYS[2]=unlock message channel
// ARGV: message/new lease/thread identifier
-- Not my lock: refuse, return nil
if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then
    return nil;
end;
-- Reentry count -1
local counter = redis.call('hincrby', KEYS[1], ARGV[3], -1);
if (counter > 0) then
    -- Not zero yet: only refresh lease
    redis.call('pexpire', KEYS[1], ARGV[2]);
    return 0;
else
    -- Zero: delete lock and broadcast wake-up
    redis.call('del', KEYS[1]);
    redis.call('publish', KEYS[2], ARGV[1]);
    return 1;
end;

Source-level key points: Unlock verifies hexists first — lock already transferred, silently returns nil instead of wrong delete; Reentry count reaches zero before del — inner not exited, outer unlock only refreshes lease; publish broadcasts unlock message — all waiters subscribed to channel wake immediately and re-contend, avoiding fixed-interval polling waste.

RedLock Controversy: A Debate With No Winners

Background: Single-node Redis lock has structural flaw — master-replica replication is async. Client acquires lock on master, master crashes before sync, replica promoted, lock gone, second client acquires successfully. RedLock (Red Lock) born to address this:

// RedLock flow (N = 5 independent masters)
// 1. Sequentially lock 5 nodes,
//     same key and random value each
// 2. Per-node lock timeout small (e.g., 50ms),
//     fail fast to next
// 3. Success count >= 3 (majority)
//     AND total time < lock validity → success
// 4. Actual validity = initial TTL - total lock time
// 5. On failure, unlock all nodes

5.1 Martin Kleppmann's Attacks

DDIA author 2016 article "How to do distributed locking":

GC Pause Attack: Client 1 acquires lock, then long GC (or process pause), lock expires and released, client 2 acquires lock and writes, client 1 GC recovers and continues writing unaware — mutex broken, RedLock cannot save it because it solves "lock service unreliable", not "lock holder unreliable".

Clock Jump Attack: RedLock validity relies on each node's local clock. A node's clock jumps forward, prematurely deems lock expired, mutex similarly broken. Majority algorithms (e.g., Paxos) correctness never depends on clocks.

His Prescription: Either sacrifice efficiency for consensus (ZK/etcd), or add fencing token — each lock acquisition returns monotonically increasing number, storage layer rejects writes with stale numbers, making "occasionally failing lock" no longer break correctness.

5.2 Antirez's Response & Industry Consensus

Redis author antirez replied same year: clock jumps mitigable via ops constraints (disable NTP jumps, use monotonic clock); GC pause exists in ZK locks too, fencing token implementable on Redis ( INCR generates sequence). But he acknowledged: RedLock's complexity yields limited gain.

Industry Practice Consensus: ① Redisson's RedLock implementation deprecated since 3.16+; ② Vast majority use "single master + watchdog" lock, accept tiny failure probability, fallback to business idempotency/version numbers; ③ True financial-grade mutex must not rely on lock itself — use DB optimistic lock/unique constraint as final defense, lock is merely performance optimization. The debate's real legacy: Distributed lock correctness can never rest solely on the lock.

Production-Grade Encapsulation: Annotation-Based Distributed Lock

Raw RLock boilerplate scatters everywhere. Combine with AOP knowledge to encapsulate as annotation (SpEL for key, echoing @Cacheable style from caching article):

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface DistLock {
    String key(); // SpEL, e.g. "order: + #orderId"
    long waitTime() default 3;
    // -1 means omit = enable watchdog
    long leaseTime() default -1;
}

@Around("@annotation(distLock)")
public Object around(ProceedingJoinPoint pjp, DistLock distLock)
    throws Throwable {
    String key = parseSpEL(distLock.key(), pjp);
    RLock lock = redisson.getLock(key);
    long lease = distLock.leaseTime();
    boolean locked = lease < 0
        ? lock.tryLock(distLock.waitTime(), TimeUnit.SECONDS)
        : lock.tryLock(distLock.waitTime(), lease, TimeUnit.SECONDS);
    if (!locked) {
        throw new LockAcquireException(key);
    }
    try {
        return pjp.proceed();
    } finally {
        if (lock.isHeldByCurrentThread()) {
            lock.unlock();
        }
    }
}

Two notes: ternary in tryLock better split into private method for clarity, compressed here for space; Lock key granularity must be fine — "order:1001" mutexes only this order, not entire order endpoint. Community already has Lock4j, ShedLock (echoes scheduled task article) — evaluate before reinventing.

Eight High-Frequency Pitfalls

Pitfall 1: Passing leaseTime disables watchdog. tryLock(3, 30, SECONDS) explicit lease stops renewal; business timeout → lock expires early. Default -1 enables watchdog.

Pitfall 2: Unlock after expiry throws exception. Business exceeds lease, lock transferred, direct unlock throws IllegalMonitorStateException</strong>. In <code>finally, check isHeldByCurrentThread() first.

Pitfall 3: Master-replica failover loses lock. Master writes, crashes before sync, new master lacks lock. High-demand scenarios use Redisson sentinel-cluster mode (partial optimizations) or switch to ZK/etcd.

Pitfall 4: Fixed value without token. Release without holder verification; A times out, deletes B's lock, entire mechanism fails. Unique token + Lua verification is baseline.

Pitfall 5: Coarse lock granularity. Locking "order endpoint" instead of "specific product/order" artificially serializes throughput. Key must contain business primary key.

Pitfall 6: Blocking lock() without timeout. Redis hiccup → threads wait indefinitely, pool exhausted. Production always tryLock + waitTime + fast-fail fallback.

Pitfall 7: Treating RedLock as silver bullet. Five nodes cross-datacenter still risk dual-hold under partition; official impl deprecated. Correctness via idempotency and fencing, not more complex locks.

Pitfall 8: Calling remote APIs inside lock. Nested slow RPC makes hold time uncontrollable, amplifies pitfalls 1/2. Compute first then lock, or move remote calls outside lock.

Troubleshooting Three Steps

Step 1: HGETALL lock key — is field your thread ID, how much TTL left, determine "never acquired" vs "acquired then lost".

Step 2: PUBSUB NUMSUB redisson_lock__channel: check subscribers; waiting queue buildup signals contention hotspot expanding.

Step 3: Business logs + thread dump align timeline: lock acquire point, release point, GC logs three-point correlation, pinpoint timeout, transfer, or GC pause.

Learning Path & Interview Questions

8.1 Four-Stage Roadmap

Beginner (1 week): Hand-write SET NX EX + Lua release lock, understand atomicity, unique token, anti-accidental-delete trio.

Intermediate (2 weeks): Read through RedissonLock three Lua scripts; grasp watchdog activation condition and renewal interval; understand pub/sub wait-wake mechanism.

Advanced (1 month): Deep-read Martin vs antirez original debate; understand fencing token and consensus systems (ZK ephemeral sequential nodes, etcd lease) similarities/differences.

Practice (ongoing): Build team annotation lock component; design "lock + idempotency" double insurance for flash sale/reconciliation; load-test lock contention hotspots and fallback paths.

8.2 Twelve Interview Questions

setnx

vs set nx ex difference? Why single command?

Why must unlock use Lua? Why not get + delete?

When does watchdog start? What is renewal interval?

Business timeout, lock expires, unlock errors — how to prevent?

How does Redisson implement reentrancy? Data structure?

How is

tryLock

waitTime

waiting implemented? Polling?

Why does master-replica failover lose lock? How to mitigate?

RedLock acquisition flow? Success conditions?

Martin's two attacks on RedLock?

What is fencing token? What problem does it solve?

How to choose Redis lock vs ZK lock? Respective scenarios?

Why "lock is performance optimization, idempotency is bottom line"?

8.3 One-Page Cheat Sheet

Lock trio: setIfAbsent + unique token
+ Lua atomic release
Watchdog three rules: no leaseTime → start
30s lease renew every 10s
Crash auto-release prevents deadlock
Unlock two steps: isHeldByCurrentThread
then unlock (reentry to 0 then del)
RedLock one line: majority 5 choose 3,
Pierced by GC and clock,
Official deprecated → single master+watchdog
Ultimate bottom line: lock = perf optimization,
Idempotency / optimistic lock = correctness safety net

Summary: Distributed lock starts from one line SET NX, passes through atomic release, watchdog renewal, reentrancy and wait-wake, finally hits RedLock debate's philosophical watershed: mutex always has theoretical breach window. Engineering answer: "Lock for performance, idempotency for correctness." Next battle enters distributed transaction finale: Spring Boot Distributed Transaction — Seata AT Mode vs TCC Handwritten .