Implementing a Redis‑Based Distributed Lock to Ensure Data Consistency in a CMDB System

In a CMDB system, machine data originates from multiple sources and is treated as streaming data that must be merged into a single, consistent record. When multiple threads read and write the same machine entry concurrently, race conditions cause lost updates, as illustrated by a series of timeline examples.

The core issue is the lack of mutual exclusion during the merge operation. The article evaluates three categories of synchronization mechanisms: thread‑level locks (Python threading.Lock), process‑level locks ( multiprocessing.Lock), and distributed locks required for multi‑machine, multi‑process deployments.

Given the CMDB’s multi‑machine, multi‑process architecture, a distributed lock is chosen. Three common implementations are compared: database‑based locks, cache‑based locks (Redis), and Zookeeper‑based locks. Redis is selected for its in‑memory speed, TTL support, and simplicity.

The design adds a lock layer between the data‑processing component and the database. A Redis key (e.g., lock_<em>machine_id</em>) acts as the lock token. The article walks through several implementation attempts:

Initial non‑atomic lock using GET followed by SET, which fails under contention.

Improved version using SETNX to acquire the lock atomically.

Adding a TTL to the lock to avoid deadlocks when a thread crashes.

Ensuring the lock creator is the only one that can release it by storing a unique value (IP‑PID‑thread‑timestamp) and deleting the lock only if the stored value matches.

Final robust version that generates a unique lock value, acquires the lock with SET (using nx=True and ex=timeout), and releases it atomically via a Lua script.

Representative Python code snippets are shown below:

def increase(redis, lock, key):
    lock_value = lock.get_lock(key)
    value = redis.get(key) or 0
    time.sleep(0.1)
    value += 1
    redis.set(key, value)
    lock.del_lock(key, lock_value)

class RedisLock(object):
    def __init__(self, rediscli):
        self.rediscli = rediscli.master
        self.ip = socket.gethostbyname(socket.gethostname())
        self.pid = os.getpid()
    def gen_unique_value(self):
        thread_name = threading.current_thread().name
        return f"{self.ip}-{self.pid}-{thread_name}-{time.time()}"
    def get(self, key, timeout=3):
        lock_key = f"lock_{key}"
        unique = self.gen_unique_value()
        while True:
            if self.rediscli.set(lock_key, unique, nx=True, ex=timeout):
                return unique
            time.sleep(0.01)
    def delete(self, key, value):
        lock_key = f"lock_{key}"
        script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end"
        self.rediscli.eval(script, 1, lock_key, value)

Extensive testing with multiple threads demonstrates that the Redis‑based lock prevents lost updates, handles thread crashes via TTL, and avoids accidental lock deletions by verifying the unique lock value. However, the article notes that if the lock timeout is shorter than the business processing time, race conditions may reappear, and that Redis master failover can introduce subtle consistency issues, suggesting the Redlock algorithm as a possible mitigation.

In summary, the Redis distributed lock provides mutual exclusion, dead‑lock avoidance through TTL, and safety against accidental lock removal, making it a practical solution for ensuring data consistency in high‑concurrency backend services such as CMDB platforms.