Mastering Failure‑Oriented Design: Mindset, Process, and Distributed Locks

Dao – The Mindset

Failure is inevitable in software systems; hardware ages, software becomes outdated, traffic spikes, and bugs appear. Designing for failure does not eliminate it but mitigates its impact on the business and the individual engineer. The author argues that beyond architecture, project management, and technical leadership, the ability to design for failure is a decisive skill differentiating senior engineers.

Key attitudes include avoiding hard‑coded assumptions, isolating variability, and regularly regressing code to prevent decay. Engineers should treat every change as a potential source of failure and build muscle memory for defensive coding.

Shu – The Process

The article outlines a robust development lifecycle that embeds failure‑oriented thinking at each stage:

Requirement stage: Perform compliance, anti‑fraud, and security assessments early to eliminate risky requirements.

Implementation stage: Design solutions with kill‑switches, fallback paths, and unit tests; keep configurations flexible to accommodate changing product needs.

Testing stage: Developers conduct self‑tests, then testing engineers perform functional and security testing, followed by broader regression testing.

Release stage: Use gray‑release strategies, gradually rolling out to subsets of machines or regions, and verify stability before full deployment.

Verification stage: Conduct post‑release online regression, load testing, and pre‑event rehearsals for high‑traffic activities.

Operation stage: Implement monitoring, alerting, and AB testing to measure impact.

Incident stage: Prioritize rapid recovery, then root‑cause analysis and systematic post‑mortems.

Organizational structures that support this approach include safety production groups, cross‑functional pair programming, and dedicated testing, test‑development, risk‑control, and security‑compliance engineers.

Ji – The Technical Details

Failure‑oriented design must be reflected in concrete code. The article presents a six‑level taxonomy of Redis distributed locks, each offering increasing guarantees of mutual exclusion, dead‑lock avoidance, and consistency.

Level 1 – Simple SetNX

redis.SetNX(ctx, key, "1")
defer redis.Del(ctx, key)

Provides mutual exclusion but no dead‑lock protection.

Level 2 – SetNX with expiration (Lua for atomicity)

redis.SetNX(ctx, key, "1", expiration)
defer redis.Del(ctx, key)

Prevents dead‑locks but cannot guarantee consistency after a master failover.

Level 3 – Random value + Lua delete

redis.SetNX(ctx, key, randomValue, expiration)
// Lua script to delete only if value matches
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end

Improves consistency by ensuring only the lock owner can release the lock.

Level 4 – Wrapper function with error handling

func myFunc() (errCode *constant.ErrorCode) {
    errCode := DistributedLock(ctx, key, randomValue, LockTime)
    defer DelDistributedLock(ctx, key, randomValue)
    if errCode != nil { return errCode }
    // doSomething
}

func DistributedLock(ctx context.Context, key, value string, expiration time.Duration) (errCode *constant.ErrorCode) {
    ok, err := redis.SetNX(ctx, key, value, expiration)
    if err != nil { return constant.ERR_CACHE }
    if !ok { return constant.ERR_MISSION_GOT_LOCK }
    return nil
}

Encapsulates lock acquisition and release with clear error codes.

Level 5 – Lease renewal (watch‑dog)

// Lua script for CAS‑based renewal
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("expire", KEYS[1], ARGV[2])
else
  return 0
end

Extends lock lifetime safely when the protected operation runs longer than the original lease.

Level 6 – RedLock or WAIT command

RedLock achieves higher consistency by requiring a majority of independent Redis nodes to acquire the lock; the WAIT command blocks until writes are replicated to a configurable number of replicas, offering a cheaper consistency guarantee.

Additional technical recommendations include rate limiting, circuit breaking, multi‑region active‑active deployments, avoiding single points of failure, and automating repetitive tasks through platformization, tooling, and automation.

Takeaways

Effective failure‑oriented design blends a proactive mindset, disciplined processes, and concrete engineering patterns such as multi‑level distributed locks. By treating failure as a first‑class design concern, teams can protect both the organization’s assets and individual engineers’ careers.