Distributed Locks and Idempotency: Principles, Implementations, and the Cerberus Solution

As internet traffic and data volumes grow, traditional centralized architectures struggle to meet high‑concurrency and massive‑data‑processing demands, prompting a shift toward distributed systems.

Distributed systems consist of loosely coupled servers and exhibit characteristics like scalability, high reliability, high concurrency, and cost‑effectiveness, but they also introduce complexities such as clock inconsistency and Byzantine failures.

Two fundamental problems arise: mutual‑exclusion (resource contention) and idempotency (preventing duplicate operations). The article illustrates these issues with examples where concurrent updates to a shared variable or task queue lead to inconsistent results.

In multi‑threaded Java environments, mutual‑exclusion is typically achieved with ReentrantLock or the synchronized keyword. ReentrantLock uses a CAS‑plus‑CLH‑queue algorithm, supporting both fair and non‑fair modes, while synchronized relies on monitorenter/monitorexit bytecode instructions. The article includes excerpts of the JDK implementations:

final boolean nonfairTryAcquire(int acquires) { ... }

final boolean acquireQueued(final Node node, int arg) { ... }

In multi‑process scenarios, operating‑system semaphores provide similar mutual‑exclusion guarantees.

Distributed locks must satisfy three basic conditions: a shared storage space, a unique identifier, and at least two states (locked/unlocked). Common external storage options include databases, Redis, Tair, MongoDB, and Zookeeper.

Typical implementations:

Zookeeper: Create an EPHEMERAL_SEQUENTIAL node; the client with the smallest sequence number holds the lock. Watch the predecessor node to avoid herd effects.

Redis: Use SETNX (or GETSET ) with an expiration to acquire a lock, handling dead‑locks via timeouts.

Tair: Similar to Redis but leverages server‑side timestamp checks to avoid clock skew.

These approaches have drawbacks such as reliance on a single external component, difficulty guaranteeing fairness, and challenges with re‑entrancy.

To address these issues, the Cerberus distributed‑lock framework was created. It supports multiple engines (Zookeeper, Tair, future Redis), offers a unified API mirroring java.util.concurrent.locks.Lock , and provides a one‑click downgrade mechanism to switch engines when the primary fails.

Cerberus also includes features like explicit lock acquisition methods ( lock() , tryLock() , unlock() ) and engine‑switching APIs ( switchEngine() ).

For idempotency, the GTIS (Global Transaction Idempotency Service) assigns a globally unique ID to each business operation using an MD5 hash of a business‑specific key. It stores the ID in Tair with SETNX , ensuring that duplicate requests are rejected. GTIS handles failure scenarios by expiring keys, using timestamps to differentiate concurrent attempts, and providing retry and fallback strategies.

Both Cerberus and GTIS have been iterated through multiple versions and are deployed in production at Meituan‑Dianping, demonstrating their practicality for large‑scale distributed environments.

In summary, solving mutual‑exclusion and idempotency in distributed systems requires careful design of lock primitives, reliable external storage, and robust fallback mechanisms; Cerberus and GTIS embody these principles and offer ready‑to‑use solutions.