Skip to content
Redis Distributed Locks: SET NX EX Pitfalls, Watchdog Renewal, and the Redlock Debate

Redis Distributed Locks: SET NX EX Pitfalls, Watchdog Renewal, and the Redlock Debate

This article analyzes three production incidents to explain Redis distributed lock pitfalls, detailing correct implementation with atomic SET NX EX, unique values, Lua unlocking, watchdog renewal for TTL dilemmas, fencing tokens for correctness guarantees, Redlock's five-step majority quorum, the Martin Kleppmann vs antirez safety debate, and a decision tree for choosing between single-instance Redis, Redlock, ZooKeeper, or etcd.

Architecture Pony
Architecture Pony
Architecture Pony

Three Production Incidents Revealing Lock Pitfalls

The article opens with three real-world accidents that illustrate fundamental distributed lock mistakes:

Double refund due to lock key design: Two concurrent refund requests each succeeded in SETNX because the lock key included a timestamp (orderId + timestamp), so they acquired different keys. The lock appeared to work but mutual exclusion never occurred. Lesson: Lock keys must be determined by the resource, not the holder.

Batch job overran TTL: A scheduled task used SET NX EX 300; data volume doubled and the job ran 420 seconds. The lock expired at 300s, another instance acquired it, and both wrote the same accounts. Increasing TTL to 3600 caused a deadlock when the holder was OOM-killed — the lock stuck for an hour. Lesson: Static TTL cannot solve the dilemma; a structural solution is needed.

Lock lost during master failover: In a Redis master-replica setup, SET succeeded on the master but hadn't replicated when the master crashed. Sentinel promoted a replica that lacked the lock; a second request acquired the lock on the new master, causing a double deduction. Lesson: Locks are efficiency tools, not correctness guarantees. If lock loss equals money loss, correctness must be enforced at the resource side (e.g., DB conditional updates).

Technical Analysis: Five Gates a Correct Lock Must Pass

1.1 From SETNX to SET NX EX: Atomic Acquisition

Old SETNX + EXPIRE are two commands; a crash between them creates a permanent lock. Since Redis 2.6.12, SET key <uuid> NX PX 30000 atomically combines "set if not exists" with a millisecond TTL. Any code splitting SETNX and EXPIRE is an accident waiting to happen.

1.2 Lock Value Must Be Unique

Using a fixed value (e.g., SET key 1) and DEL on unlock cannot distinguish your lock from another's. Correct pattern: SET key <requestId> NX PX ... where requestId = instanceId + threadId + random. Unlock must use a Lua script that atomically checks the value and deletes only if it matches — GET + DEL separately leaves a window where the lock expires, another client acquires it, and your DEL removes their lock.

1.3 TTL Dilemma: Watchdog Renewal Is the Only Structural Fix

TTL too short → lock expires before business finishes; TTL too long → failure recovery waits too long. The solution is a watchdog : after acquiring the lock, start a background task that every TTL/3 (e.g., 10s for a 30s TTL) runs a Lua script to verify the lock value is still yours and reset the TTL to full. If the process dies, heartbeats stop and the lock auto-expires. Three critical points:

Renewal must be atomic (Lua: check value + PEXPIRE).

Renewal failure must alert and immediately stop the protected operation (Redisson throws an exception).

GC pauses are the watchdog's enemy: a 10s STW prevents renewal, the lock expires, another client enters, and the original process resumes believing it still holds the lock — the core scenario Martin Kleppmann's fencing addresses.

1.4 Master-Replica Failover: Single-Instance Lock Ceiling

SET is a write command landing on the master; replication is asynchronous. If the master crashes before replicating the SET, the promoted replica lacks the key, and a second client acquires the lock. Redlock was designed to close this gap: use N (typically 5) fully independent masters (no replication), send SET to all, and require a majority (≥3) to succeed. A single instance failure loses at most one vote, preserving the majority.

However, Redlock does not solve client-side pauses (long GC, I/O stalls). If the holder pauses after acquiring a majority, the lock expires, another client gets a new majority, and two holders coexist. This problem is identical for single-instance and Redlock because the pause occurs on the holder side. The defense is fencing tokens at the resource side.

1.5 Fencing Tokens: The Last Line of Correctness

Each lock grant returns a monotonically increasing token. The client presents the token when accessing the protected resource; the resource rejects any request with a token smaller than the maximum it has seen. Example: Client A gets token 33, GC pauses 20s; lock expires, B gets token 34 and writes; A wakes and submits with 33 → resource rejects because 33 < 34. Practical implementations: DB version column (optimistic locking), monotonic sequence numbers, or DB conditional updates (UPDATE ... WHERE status='PENDING'). Core insight: Redis locks (including Redlock) provide efficiency — avoiding duplicate work; correctness — never wrong — must be guaranteed by atomic conditions or versioning at the resource side.

1.6 Redlock Five-Step Procedure and Two Costs

Record start time T1; send SET NX PX to 5 instances sequentially with a small connection timeout (e.g., 50ms).

If ≥3 succeed, record end time T2.

Verify total elapsed (T2-T1) < lock validity; otherwise treat as failure.

Actual validity = validity - (T2-T1).

On failure, send DEL to all instances (including those not acquired) to clean up half-acquired locks. Unlock similarly uses Lua verify-and-delete on all instances.

Costs: Latency — 5 serial network round-trips (even parallelized, an order of magnitude slower than single-instance). Operations — 5 independent deployments, monitoring, capacity; they must remain completely independent (no replication, no clustering). If anyone "saves effort" by making them master-replica, Redlock's safety foundation collapses. The intersection of scenarios needing cross-process mutex, tolerating total Redis failure, and refusing ZooKeeper/etcd is very small.

1.7 Martin vs. antirez: What They Actually Disagreed On

Martin Kleppmann (2016) argued that client pauses + clock jumps can make Redlock produce two holders, so it provides no correctness guarantee — only an efficiency tool; for correctness use fencing tokens or consensus systems (ZooKeeper/etcd). antirez replied that pauses can be mitigated with long TTLs + resource-side checks, clock jumps avoided by operational practices, and sketched token implementations. Both are right; the divergence is the definition of "safety": Martin demands mathematical safety under all faults; antirez accepts engineering "extremely low probability + mitigable". The engineer's takeaway: never expect any lock to give you correctness; build correctness at the resource side with conditional updates. Make that a team iron law and the debate becomes irrelevant.

Core Processes: Complete Lock-Acquire-Renew-Release Sequences

2.1 Standard Acquire-Execute-Release (Refund Example)

T1: Request A executes SET lock:refund:1001 <uuidA> NX PX 30000 → success.

T2: Start watchdog (renew every 10s).

T3: Execute business — check order status (idempotent pre-check), DB conditional update for refund.

T4: Request B arrives, SET returns nil, retries with 200ms interval, exponential backoff, total timeout 3s.

T5: A finishes, stops watchdog, Lua verifies value is uuidA then DEL.

T6: B's next retry acquires lock, executes business. Key: The DB conditional update at T3 ensures that even if any mishap occurred before T5 (A crashes, lock drifts), the refund cannot execute twice. Redis lock merely keeps duplicate requests out the door.

2.2 Three Strategies on Lock Acquisition Failure

Fast fail — request is droppable (e.g., scheduled job sees another running), return "operation in progress".

Limited retry — request must complete (e.g., payment callback), exponential backoff with total timeout, then fallback to a degradation path (push to MQ for delayed processing), never infinite retry occupying threads.

Queue blocking — strong serial requirement (e.g., inventory deduction), but Redis lock retry is polling: low throughput, thundering herd. True queuing should use MQ serial consumption or ZK sequential nodes, not forcing Redis locks.

The difference between accidents is whether the failure branch was designed.

2.3 Watchdog Full Lifecycle

Lock acquired → register renewal task (Netty/scheduler timing wheel), interval TTL/3.

Each renewal → Lua "GET compare uuid + PEXPIRE reset"; success renews full TTL, failure → immediate loss-of-lock awareness .

Loss-of-lock awareness → cancel future renewals, mark business thread (interrupt/exception), stop protected operation at nearest safe point.

Business complete/error → cancel renewal task, proceed to unlock.

Process death → nothing happens; this is by design: no heartbeat, lock expires naturally within TTL.

Three common mistakes: renewal interval too tight (Redis pressure); after loss-of-lock, continuing to run "pretending to hold lock"; finally block must cancel watchdog and unlock (only unlocking leaves renewal task trying to renew a key now owned by another — Lua blocks it but creates needless Redis calls and alert noise).

2.4 Reentrant Locks: Why Hash Structure Is Required

Reentrancy means the same holder (same instance + same thread) can acquire the same lock multiple times without deadlock — needed for recursive calls, layered aspects. In Redis, String cannot support this; Hash must be used : field = holder identifier, value = reentry count. Acquire Lua: if not exists → HSET count=1 + PEXPIRE; if exists and field is self → HINCRBY +1 and refresh TTL; else fail. Release Lua: if field is self → decrement count, delete key only when count reaches 0. Reentrancy is per-holder, not global: different instances have separate fields, mutual exclusion still holds. Redisson's reentrant lock uses exactly this; drawing the Hash structure and two Lua branches in an interview wins 80% of candidates.

Core Source Code: Five Key Implementations

3.1 Lock/Unlock Lua: Source of Atomicity

-- Lock: NX + PX in one command
SET lock:refund:1001 <uuid> NX PX 30000
-- Returns OK on success, nil on failure

-- Unlock: verify holder + delete, must be atomic
if redis.call("GET", KEYS[1]) == ARGV[1]
then
    return redis.call("DEL", KEYS[1])
else
    return 0  -- not your lock, refuse to delete
end
-- Two pitfalls: GET+DEL separate has window;
-- failure returns 0 not exception, so caller
-- can distinguish "lock already transferred" from "Redis failure"

3.2 Watchdog Renewal Lua: Verify + Reset in One

-- Renewal: verify holder + PEXPIRE atomically
if redis.call("GET", KEYS[1]) == ARGV[1]
then
    return redis.call("PEXPIRE",
    KEYS[1], ARGV[2])  -- renew to full TTL
else
    return 0  -- loss-of-lock signal!
end
-- Caller semantics:
-- return 0 → immediately cancel subsequent renewal tasks
-- → mark business as "lock lost"
-- → protected operation stops at safe point ASAP

3.3 Reentrant Lock Dual Lua: Hash Counter Version

-- Acquire (reentrant): KEYS[1]=lock key
-- ARGV[1]=holder ARGV[2]=TTL ms
if redis.call("HEXISTS", KEYS[1],
    ARGV[1]) == 0
then
    if redis.call("EXISTS",KEYS[1]) == 0
    then
        redis.call("HSET",KEYS[1],
        ARGV[1],1)
        redis.call("PEXPIRE",KEYS[1],
        ARGV[2])
        return 1
    end
    return 0  -- someone else's lock
end
redis.call("HINCRBY",KEYS[1],
    ARGV[1],1)      -- reentry +1
redis.call("PEXPIRE",KEYS[1],ARGV[2])
return 1

3.4 Redisson Watchdog Skeleton: Lifecycle Management

RLock lock = redisson.getLock(key);
try {
    // Omitting leaseTime enables watchdog!
    // Default TTL 30s, renew every 10s
    boolean ok = lock.tryLock(3, 5,
    TimeUnit.SECONDS);
    // Give up if not acquired within 3s
    if (!ok) throw new BusyException();
    doBusiness();  // protected operation
} finally {
    // Only unlock if still held by current thread
    if (lock.isHeldByCurrentThread()) {
        lock.unlock();
    }
}
// Passing leaseTime disables watchdog —
// if you use watchdog, don't specify leaseTime

Two biggest Redisson pitfalls hide here: (1) passing leaseTime disables the watchdog (it assumes you explicitly set the lease) — cause of most "lock expired before business finished" recurrences; (2) unlock must be guarded by isHeldByCurrentThread() — after business timeout the lock may have expired and been re-acquired, direct unlock throws IllegalMonitorStateException or worse, deletes the new owner's lock in some versions. Third pitfall: waitTime=0 semantics — it's not "no wait"; some versions degrade to blocking lock, showing as all threads stuck during stress tests; must verify empirically.

3.5 Fencing Token in Practice: DB Conditional Update

-- Refund idempotency: state machine is fencing
UPDATE refund_order
SET  status = 'REFUNDED',
     operator = ?
WHERE order_id = ?
  AND status = 'PENDING';
-- affected rows = 0
-- → already processed (or processing)
-- → return success directly (idempotent)

-- Inventory oversell prevention: same pattern
UPDATE stock
SET  num = num - ?
WHERE sku_id = ? AND num >= ?;

These two SQL statements are the minimal implementation of "correctness fence at the resource side": regardless of what happens to the Redis lock — expiry, drift, accidental deletion — the second request reaching the DB will always see affected rows = 0 . Engineering discipline: every time you write a Redis lock, ask "when the lock is pierced, what DB mechanism stops the second request?" If you can't answer, the lock isn't deduplication — it's a placebo.

Advanced Topics: Six Accounts You Must Calculate Upfront

4.1 Lock Granularity: Finer = More Concurrency, Finer = More Danger

Granularity directly determines throughput: locking "all inventory" vs. "single SKU" are orders of magnitude apart. Three refinement levels: Resource-level (lock:sku:1001 not lock:stock), Operation-level (reads usually don't need locks — reads go to cache, DB has its own consistency), Stage-level (long flows split into short critical sections; lock only the few lines that truly need mutual exclusion; network calls and computation moved outside). RPC inside critical section is the most common design pollution : holding lock while waiting 2s for remote response makes watchdog renew frantically, all other instances queue up. Fix: move RPC outside — inside lock do "pre-reserve", outside call, inside lock "confirm or rollback". Lower bound warning: inconsistent acquisition order of multiple fine locks = deadlock; multiple locks must always be acquired in global sorted order.

4.2 When Single-Instance Lock Is Enough: Scenario Criteria

Most business locks really need "don't repeat work" — repetition is waste, not catastrophe, because DB idempotency catches it. This scenario: single-instance Redis lock + watchdog + DB conditional update is a complete solution ; Redlock is pure overhead. True need for Redlock-level strong mutex: resource cannot be guarded by conditional update (external side-effects: bank API, filesystem); repetition is irreversible . But such scenarios should first ask "should we use Redis lock at all?" — ZooKeeper/etcd consensus locks provide built-in fencing, session-based auto-release, watch notifications, protocol-level safety. Cost: write throughput lower, operational surface larger. One-sentence criterion: DB can fence → single-instance Redis lock; DB cannot fence → consider ZK/etcd first, then Redlock.

4.3 Clock Issues: TTL's Hidden Assumptions

Redis PX expiry is decided by the Redis server clock ; client clock doesn't affect the lock itself. But Redlock semantics require the client to "calculate remaining validity after acquiring lock", which implicitly assumes node clocks are roughly synchronized and don't jump — NTP step changes can cause premature or delayed expiry, exactly Martin's "clock assumption unsafe" point. Engineering defenses: disable server clock step changes (use slew for smooth sync); leave headroom in validity calculation (abort if elapsed exceeds half validity); replace absolute time with monotonic clocks or version numbers for all correctness-critical timing. Clocks are the least trustworthy infrastructure in distributed systems; any design whose correctness depends on clocks deserves a question mark.

4.4 Lock Keys vs. Cache Keys: Namespace Isolation

Frequent low-level accident: lock keys and data keys share a namespace, then wildcard deletion or migration scripts delete locks. Example: ops bulk-clean product cache DEL cache:sku:*, script mistakenly written as DEL *, lock key lock:sku:1001 disappears — all instances simultaneously "acquire lock successfully". Three defenses: Lock keys use independent prefix (lock: vs cache: separated; cleanup scripts only match cache:); High-security: lock keys on separate DB or separate instance (physical isolation — cache eviction, OOM never affect locks); Cleanup patterns whitelisted — deletable patterns in config, approved, no ad-hoc handcrafting. Additional trap: Lock instance must never have maxmemory eviction configured — lock evicted by LRU = silent lock loss.

4.5 Observability & Emergency: Making the Lock System Visible

Lock problems are silent; must actively observe four metrics: (1) Acquisition failure rate — normally single-digit percent; spike = critical section lengthened or lock leak. (2) Average hold duration — continuously approaching TTL = watchdog renewing at high frequency, business duration exceeds design assumption. (3) Watchdog renewal failure count — non-zero = lock-loss event, must alert; real-time signal that mutual exclusion has been pierced. (4) Lock key TTL distribution — sampling scan finding TTL -1 (permanent locks) triggers immediate manual intervention. Two pre-staged emergency levers: admin API "force release specific lock" (with approval, for leaked permanent locks); global "lock read-only mode" switch (extreme case: skip locking, go straight to DB idempotency — better duplicate rejects than mutual deadlock). Drill: quarterly "kill lock-holder process" exercise — verify auto-expiry, peer takeover, DB idempotency no duplicates — if any of the three answers is missing, there's no safety net.

4.6 Selection Decision Tree: Redis Lock / Redlock / ZK / etcd

Collapse the entire lock selection into a tree:

Q1 Can repetition consequences be fenced by DB conditional update/idempotency? Yes → Single-instance Redis lock + watchdog (80% of scenarios stop here).

Q2 Cannot fence, need strong mutex, team already has ZK or etcd? → Use them: session keep-alive, ephemeral nodes prevent deadlock, watch notifications avoid polling, built-in fencing — more targeted than Redlock.

Q3 Cannot fence and don't want ZK/etcd? → Redlock (5 independent instances + full procedure), accepting it's still not mathematically safe, needs resource-side token reinforcement.

Q4 Just "prevent scheduled jobs from running concurrently"? → Don't even need distributed lock: ShedLock / XXL-Job scheduling deduplication suffices.

Each step right doubles operational cost; benefit is only a smaller mutex failure rate — selection is essentially pricing the cost of duplicate execution.

Learning Path: From Usage to Internals

Week 1: Hand-write a correct lock — SET NX PX + unique value + Lua unlock + Lua renewal; test mutual exclusion and renewal with two threads; deliberately break it (separate SETNX/EXPIRE, separate GET/DEL) and observe piercing.

Week 2: Read Redisson lock source — RedissonLock tryLock skeleton, watchdog RenewalTask, reentrant Hash structure; map to the Lua in sections 3.3/3.4 line by line.

Weeks 3-4: Deep-read the two original debate papers — Martin's "How to do distributed locking" and antirez's "Is Redlock safe?"; restate both arguments and the root divergence in your own words.

Deeper: Compare ZK ephemeral sequential node lock (Curator InterProcessMutex) with Redis lock; understand the philosophical split between "consensus-system locks" (protocol guarantees safety) and "heartbeat-renewal locks" (engineering mitigates risk).

High-Frequency Interview Questions (with Summary Answers)

The article lists 20 questions covering all key points. Each answer gives the conclusion first, then the reasoning. Key examples:

Q1 Minimal correct implementation: Three-piece set: atomic SET NX PX, unique requestId with Lua verify+delete, watchdog for unknown business duration.

Q2 Why SETNX+EXPIRE fails: Crash between commands → permanent lock → deadlock.

Q3 Why unlock needs Lua: GET+DEL window allows deleting another's lock after expiry+reacquire.

Q4 Lock expires but business unfinished: Watchdog renewal is the only structural fix; static TTL tuning cannot solve.

Q5 Watchdog renewal failure meaning: Lock already transferred or deleted; must immediately cancel renewals, mark business as lock-lost, stop at safe point, alert.

Q6 Master-replica loses lock: Async replication; master crashes before replicating SET; promoted replica lacks key.

Q7 Redlock flow & why 5 instances: 5 independent instances, sequential SET, majority ≥3 (2N+1 tolerates 2 failures), verify elapsed < validity, actual validity = validity - elapsed, cleanup all on failure. Instances must be fully independent.

Q8 Martin's core critique: Client pause + clock jump → two holders; Redlock gives no correctness guarantee, only efficiency; correctness needs fencing tokens or consensus locks.

Q9 Fencing token & implementation: Monotonic token per grant; resource rejects lower tokens; implement via DB version, state machine conditional update, monotonic sequence.

Q10 Reentrant lock in Redis: String → Hash; field=holder, value=count; Lua branches for first acquire vs reentry; per-holder not global.

Q11 Redisson tryLock params & misuses: waitTime/leaseTime; leaseTime disables watchdog; unlock must guard with isHeldByCurrentThread; waitTime=0 may block in some versions.

Q12 Can't get lock — three strategies: Fast fail, limited retry with fallback to MQ, true queuing via MQ/ZK not Redis polling.

Q13 Lock granularity design: Resource/operation/stage refinement; RPC in critical section is pollution; global order for multiple locks.

Q14 When ZK/etcd over Redis: External side-effects, irreversible repetition, team has ZK/etcd; consensus locks give ephemeral nodes, watch, built-in fencing.

Q15 GC pause impact & defense: STW stops watchdog → lock expires → other acquires → resume thinks still holds; defense: resource-side fencing/conditional update (root), generous TTL+renewal interval, re-verify lock state after resume.

Q16 Lock key + cache key same namespace risk: Wildcard cleanup deletes locks → all instances acquire simultaneously; defenses: separate prefix, separate DB/instance, whitelisted cleanup patterns, no maxmemory eviction on lock instance.

Q17 Lock metrics to monitor: Failure rate, avg hold time, watchdog failure count, TTL distribution; emergency: force-release API + read-only switch; quarterly kill-holder drill.

Q18 Clock impact on Redis lock: Single-instance PX uses server clock; Redlock remaining-validity calc assumes synced non-jumping clocks; defenses: slew sync, headroom in calc, monotonic clocks/versions for correctness.

Q19 Redis lock vs DB optimistic lock: Not duplicate — division of labor: Redis lock = efficiency layer (blocks 99% duplicates from hitting DB); DB conditional update/version = correctness layer (catches the 1% that pierce the lock). Both required.

Q20 How to verify lock correctness: Five test classes: concurrent mutual exclusion, renewal beyond TTL, self-healing after holder kill, forced pierce injection verifying resource-side fence, failure-branch degradation not hogging threads.

One-Page Cheat Sheet

Lock in one command: SET key uuid NX PX 30000; SETNX+EXPIRE separate = deadlock candidate.

Unlock in one Lua: GET compare uuid + DEL atomic; renewal same "verify+PEXPIRE" atomic.

One fix for the dilemma: TTL too short pierces, too long waits — only watchdog breaks it: TTL/3 renewal, failure = lock-loss awareness, process dead = heartbeat stops = auto-heal.

Redisson three traps: leaseTime passed → watchdog dead; unlock without isHeldByCurrentThread; waitTime=0 semantics need empirical test.

Redlock five steps: 5 independent instances sequential SET, majority ≥3, elapsed check, validity deduction, failure full delete — closes master-failover lock loss, does NOT close client-pause piercing.

Debate in one sentence: Martin wants mathematical safety, antirez wants engineering usability; engineer's answer: resource-side fencing (conditional update/version), lock handles efficiency not correctness.

Selection in one sentence: DB can fence → single-instance lock+watchdog; DB cannot fence → ZK/etcd first, then Redlock; just prevent cron overlap → ShedLock is enough.

Summary & Next Episode Preview

Starting from three accidents, this article builds a complete Redis distributed lock system: atomic single-command acquisition, Lua atomic unlock, watchdog renewal solving the TTL dilemma, reentrant Hash counter structure, Redlock's five-step flow and two costs, the Martin vs. antirez debate's root divergence, and the iron law: Locks are efficiency tools, not correctness guarantees — every lock you write must answer "when pierced, what stops the second request?" Next episode is the series finale — Production Practice: Big Keys, Hot Keys, and Observability : big key detection and splitting, hot key local caching and sharding, slow log and latency spike diagnosis, plus a production checklist ready to deploy.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

RedisZooKeeperRedissonRedlockwatchdogDistributed LocksLua scriptingfencing token
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.