Why Your Distributed Lock Keeps Failing in Production (And How to Fix It)

When interviewers ask "How do you implement a distributed lock?" they often start with a simple question, but quickly probe deeper: how to handle service crashes, timeout settings, and reliability in a Redis cluster. Mastering these follow‑up questions requires understanding the three core problems a lock must solve: availability (the lock service must never become a single point of failure), deadlock (the lock must be released automatically if the holder crashes), and split‑brain (multiple nodes must not believe they each hold the lock after a cluster partition).

Database‑Based Locks

Using MySQL for a distributed lock is the most straightforward idea. The pessimistic approach employs SELECT ... FOR UPDATE to lock a row, forcing other transactions to wait. This is simple but incurs heavy blocking under high concurrency and can cause circular deadlocks (e.g., transaction A waits for B while B waits for A). The optimistic approach adds a version column ver; readers fetch the version and writers update only if the version matches, retrying on mismatch. This avoids blocking but can lead to many retries when contention is high. While easy to understand, database locks suffer from poor performance because higher isolation levels reduce throughput, making them unsuitable for large‑scale flash‑sale scenarios.

Redis Distributed Locks – Five Evolutionary Steps

Redis is the de‑facto choice for high‑performance distributed locks, yet many developers stop at the basic SETNX command. The evolution includes:

SETNX : Sets a key only if it does not exist, providing mutual exclusion. However, if the client crashes after acquiring the lock, the lock never releases, causing a deadlock.

SET NX PX : Adds an expiration time in a single atomic command, preventing permanent deadlocks. The new issue is that the lock may expire before the critical section finishes, allowing another client to acquire it while the original holder still runs, potentially deleting a lock it no longer owns.

Lua script for safe release : Executes a script that first checks whether the lock value matches the client’s identifier before deleting, ensuring that only the owner can release the lock.

Watchdog (auto‑renewal) : A background thread periodically extends the lock’s TTL, eliminating the need to guess an appropriate timeout. The thread stops when the main work finishes; otherwise the lock stays valid.

Redlock (cluster voting) : To survive a single‑node failure, the client attempts to acquire the lock on N independent Redis instances. If a majority (N/2 + 1) succeed, the lock is considered acquired. For example, with five instances, at least three must grant the lock, providing redundancy.

Redlock Algorithm Details and Caveats

Redis replication is asynchronous; after a master acquires a lock, it may crash before propagating the lock to replicas, leading to a split‑brain where another replica grants the same lock. Redlock mitigates this by requiring majority agreement. However, it introduces latency because the client must wait for responses from a majority of nodes, and GC pauses on the client side can cause the lock to expire prematurely. Consequently, Redisson’s documentation advises cautious use of Redlock, and many teams prefer simpler primary‑replica waiting strategies.

Interview Answer Framework

When asked about distributed locks, structure the response as follows:

State the three core problems a lock must address: availability, deadlock, and split‑brain.

Outline four design principles: mutual exclusion, high availability, automatic expiration, and re‑entrancy.

Compare database‑based locks (pessimistic vs. optimistic) with Redis‑based locks, highlighting performance trade‑offs.

Discuss the five Redis evolution steps and the Redlock algorithm, noting its advantages and the concerns raised by Redisson.

The key is to demonstrate reasoning rather than merely reciting commands like SETNX.

One‑Sentence Takeaway

The essence of a distributed lock is not "how to lock" but "how to reliably coordinate multiple processes in an unreliable distributed environment," and mastering this concept equips you to handle any interview angle.