Skip to content
GC Pauses Break Redis Locks: How Fencing Tokens Prevent Stale Overwrites

GC Pauses Break Redis Locks: How Fencing Tokens Prevent Stale Overwrites

The article explains how Redis distributed locks with TTL leases fail to prevent stale writes when a garbage collection pause delays a lock holder past its lease expiry, allowing a new holder to write fresh data that the original holder then overwrites upon resuming; it demonstrates why fencing tokens — incrementing sequence numbers issued atomically with lock grants and checked atomically at the storage layer via conditional updates — are necessary to reject obsolete writes.

IT Architects Alliance
IT Architects Alliance
IT Architects Alliance

You added a Redis lock to a shared-configuration update endpoint and the code review passed. But what happens when the thread that acquired the lock gets paused by a stop-the-world GC, the lock expires, another instance acquires the lock and writes new data, and then the original thread resumes and overwrites that new data with its stale in-memory copy?

Acquiring a lock only grants a time-limited lease

The typical pattern uses a single atomic Redis command:

SET lock:config:42 9f27c6e1a0484db2b35584c792e806ad NX PX 30000

The hexadecimal string is a unique random value generated per request. NX requires the key not to exist; PX 30000 sets a 30-second TTL. Both happen in one command, eliminating the gap where a process could crash after setting the key but before setting the expiry. Only on success does the instance proceed to read, modify, and write back the configuration.

This lock is a lease: when it expires, Redis allows the next requester to acquire it. The random value identifies the current holder. On release, the client must atomically compare the value and delete only if it matches, preventing accidental deletion of another holder's lock.

This solves "lock held forever" and "deleting the wrong lock." However, the database receiving the update only checks connection credentials and business conditions — it has no knowledge of who holds the Redis lock. The lock key and the business data live in separate systems; lease expiry does not automatically become a write-rejection condition at the database.

GC pauses the thread, but the lease countdown keeps running

Consider this timeline (seconds from lock acquisition):

0s: Instance A acquires the lease, reads the old configuration.

5s: Stop-the-world GC pauses all application threads.

30s: Lease expires.

31s: Instance B acquires the new lock, reads the configuration, modifies it.

32s: B writes successfully.

40s: A resumes, continues the next statement, and writes its stale in-memory configuration — overwriting B's update.

Throughout, Redis correctly acknowledges only one valid holder at any moment. Yet the database accepted two sequential updates because the lease expiry only revokes eligibility in the lock service; A's thread, memory, and database connection remain alive and can still reach the write statement.

Lease handover has no overlap, yet the resumed old client's write still overwrites the new configuration
Lease handover has no overlap, yet the resumed old client's write still overwrites the new configuration

Increasing the TTL merely pushes the trigger window further out. Automatic lease renewal can cover normal long-running tasks, but if the renewal thread runs in the same process it gets paused by the same GC. After renewal fails the client should stop further work, yet already-sent requests may still be in flight. Checking the lock again just before writing leaves a gap: the check passes, then a pause occurs, then the write is sent after resumption. To close this path, the validity decision and the actual resource modification must happen in the same controlled entry point.

Switching to Redlock still requires separate handling of expired writes

Even with Redlock — which uses multiple independent Redis masters, requires a majority, and subtracts request latency plus clock-drift margin — a long pause after lock acquisition still needs to be handled at the write side.

In his response to Martin Kleppmann's "Is Redlock safe?", antirez emphasizes that if a long delay occurs during lock acquisition, the client must recalculate the remaining validity and abort if it has expired; treating a late response as a valid lock misses the algorithm's own check. Martin's "How to do distributed locking" focuses on resource correctness: even with correct lock acquisition, subsequent pauses and request latency can let stale work bypass the lease. antirez stresses system timing assumptions and the reality that some external resources cannot check tokens. Where storage already supports conditional updates, some scenarios can use that directly. The two discussions address lock-grant safety and resource protection as separate layers.

After the new holder writes, storage must block the old token

A fencing token is a monotonically increasing sequence number returned with each successful lock grant. Think of it as a generation number for resource access. The random unlock value identifies "is this me?"; the incrementing sequence identifies "is this an old generation?".

Adopt a protocol that returns an incrementing token with the grant. The original SET and Redlock do not provide this. A receives token 41, B receives 42. Every write carries its token. Storage records the highest accepted token; after accepting B's 42, it rejects A's 41.

For a unique config id with a non-null last_fence initially 0, each grant increments once. A PostgreSQL single-row conditional update:

UPDATE config
SET payload = :new_payload, last_fence = :token
WHERE id = :id AND last_fence < :token
RETURNING id;

For config id=42, B submits token 42 and succeeds (returns id=42). A then submits token 41 and gets an empty result; B's content remains.

The token check, token persistence, and payload write happen in one statement. Under Read Committed, an update encountering an uncommitted write on the same row waits; after that transaction commits, the condition is re-evaluated against the new row. An empty result tells the caller to abandon the stale computation. To retry, re-acquire, re-read, and recompute.

If 41 arrives before 42 is persisted, 41 may still succeed. Fencing guarantees that once storage accepts a new generation, it rejects older ones. Token checks alone cannot detect that a lease just expired. Resubmitting the same token is also rejected. If the success response is lost, use a business idempotency key or a result query to confirm the outcome.

Atomic check uses storage's accepted high-water mark to reject old writes, not just lease expiry
Atomic check uses storage's accepted high-water mark to reject old writes, not just lease expiry

One concrete implementation uses etcd 3.6 transactions to contend for a single lock key: create a lease, then in one transaction check that the key does not exist and, if so, write the key with the lease; the transaction's revision becomes the token.

The token state is persisted by the etcd cluster; grant and numbering are determined in the same commit. Normal restarts and leader changes preserve committed state; recovery from an old snapshot requires separate handling for token rollback.

Revisions jump (41/42 are simplified generation numbers). Running an independent INCR after a Redis lock acquisition has a flaw: if A pauses until after B finishes and then fetches a token, A gets a larger number.

Storage must persist the sequence number, and every write entry point must perform the check. If the external interface only has an idempotency key, it can deduplicate the same request; but if A resubmits the stale payload with a different request ID, it can still overwrite B. If the interface supports conditional updates by resource version — B changes version from v1 to v2 — then A's update carrying the old version v1 is rejected. When the interface checks neither generation nor resource version, locking and request deduplication still cannot prevent this stale-content overwrite.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

redisPostgreSQLdistributed lockingGC pauseetcdRedlockfencing tokensstale writes
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.