Master Distributed System Design: Patterns, Performance & Fault Tolerance

Design Patterns

Gateway

Functions

Request routing: clients call the gateway, which forwards requests to registered services.

Service registration: back‑end services register APIs, and the gateway maintains routing tables.

Load balancing: supports round‑robin, random, weighted, session‑sticky, etc.

Security: HTTPS, authentication, DDoS protection.

Canary/gray releases: target specific service versions or tenants.

API aggregation: combine multiple back‑end APIs to reduce client calls.

API orchestration: chain APIs to implement business logic.

Design considerations

High availability and fault tolerance.

Scalable architecture to support business‑specific flow control.

High performance using asynchronous I/O frameworks (e.g., Java Netty, Go channels).

Strong security (encryption, authentication, DDoS mitigation).

Operational visibility: monitoring, capacity planning, auto‑scaling.

Decoupled business logic; plug‑in mechanisms; deploy gateways close to back‑ends to minimize latency.

Sidecar

Value

Separates control plane (routing, flow control, circuit breaking, idempotency, service discovery, auth) from business logic.

Applicable to legacy migration, multi‑language microservices, and multi‑provider environments.

Design points

Use language‑agnostic service protocols.

Aggregate control functions (flow control, circuit breaking, retries, idempotency) to keep business code simple.

Avoid intrusive IPC (signals, shared memory); prefer local network protocols such as TCP or HTTP.

Service Mesh

The service mesh provides an infrastructure layer for inter‑service communication.

Key characteristics

Application‑level communication middle‑layer.

Lightweight network proxy (sidecar) per service instance.

Decouples application code from transport concerns.

Applications remain unaware of the mesh.

Main frameworks

Istio

Linkerd

Distributed Lock

Solutions

Redis lock: SETNX key value PX expireTime. value must be globally unique (e.g., UUID, TraceID, /dev/urandom output). expireTime is in milliseconds; the lock auto‑releases after timeout.

Pessimistic lock: acquire lock before operation; low throughput.

Optimistic lock: version‑based compare‑and‑set; high throughput, suitable for read‑heavy workloads.

CAS (compare‑and‑swap) can replace a lock for simple shared‑resource updates.

Design points

Exclusivity: only one client may hold the lock.

Automatic timeout‑based release.

High availability and persistence of lock state.

Non‑blocking and re‑entrant.

Avoid deadlocks; ensure every lock can be released.

Cluster fault tolerance: lock remains usable despite node failures.

Configuration Center

Static configuration: environment variables and startup parameters.

Dynamic configuration: runtime adjustments such as flow‑control switches or circuit‑breaker toggles.

Asynchronous Communication

Request‑response: sender directly invokes receiver.

Polling: sender periodically queries receiver.

Callback: sender registers a callback endpoint; receiver invokes it after processing.

Event‑driven (EDA): publish/subscribe via a broker (e.g., RocketMQ) to decouple producers and consumers.

Benefits: service decoupling, improved isolation, and better resilience.

Idempotency

Ensures repeated execution of an operation yields the same result. Core techniques include:

Globally unique identifiers: database auto‑increment IDs, UUIDs, Redis‑generated IDs, Snowflake algorithm.

Idempotent HTTP methods: GET, HEAD, PUT, DELETE, OPTIONS (POST is non‑idempotent).

Performance

Distributed Cache

Cache‑Aside: application explicitly manages cache reads, writes, and invalidation.

Read/Write‑Through: cache acts as a proxy, synchronously updating the database.

Write‑Behind (Write‑Back): writes are buffered in cache and flushed to the database asynchronously.

Asynchronous Processing

Push model: a central scheduler pushes tasks to workers (higher complexity).

Pull model: workers pull tasks from a queue (simpler).

Hybrid push‑pull for flexible workloads.

Database Sharding

Vertical sharding: split tables by column groups.

Horizontal sharding: split rows across shards using hash or time‑range algorithms.

Design tips: reserve identifier space for future shards, parallelize shard aggregation, avoid cross‑shard transactions, keep business logic shard‑agnostic.

Fault Tolerance

System Availability

Key metrics:

MTTF (Mean Time To Failure) – longer is better.

MTTR (Mean Time To Recover) – shorter is better.

Availability = MTTF / (MTTF + MTTR).

Service Degradation

Reduce consistency guarantees (e.g., switch from strong to eventual consistency).

Disable non‑essential services or features to free resources.

Simplify business processes to lower inter‑service communication overhead.

Rate Limiting

Purpose

Guarantee SLA.

Handle traffic spikes and reduce capacity‑planning costs.

Tenant isolation to prevent a single user from exhausting shared resources.

Algorithms

Counter‑based limits (per‑user or per‑service counters).

Queue‑based limits: FIFO, weighted queues, token bucket.

Dynamic flow control: adjust allowed QPS based on real‑time latency.

Design tips

Provide a manual emergency switch.

Integrate monitoring and alerting for limit breaches.

Return user‑friendly error codes.

Propagate rate‑limit identifiers in RPC traces for downstream handling.

Circuit Breaker

Three states:

Closed – normal operation; error count is monitored.

Open – all requests are rejected immediately.

Half‑Open – limited traffic is allowed to test recovery.

Design considerations: define error thresholds that trigger breaking, uniform logging, automatic diagnostics and recovery, optional manual toggle, and isolate circuit breaking per business domain.

Compensation Transactions

Address CAP trade‑offs (Consistency, Availability, Partition tolerance) and BASE principles (Basic Availability, Soft state, Eventual consistency). Typical strategies include exponential backoff retries and designing idempotent compensation steps.

DevOps

Deployment

Infrastructure: public cloud, private cloud, hybrid cloud.

Container platforms: Docker, Kubernetes.

Deployment strategies

Stop‑the‑world (full downtime).

Rolling updates.

Blue‑green deployments.

Canary releases.

A/B testing.

Configuration Management

Ansible

Puppet

Shippable

Monitoring

Nagios

DynaTrace

CI/CD

Continuous integration tools such as Jenkins and CodeShip; pipelines automate build, test, and delivery stages.

Engineering Efficiency

Agile Management

Scrum framework for iterative development.

Continuous Integration

Jenkins and CodeShip automate code integration and testing.

Continuous Delivery

Automated release pipelines enable frequent, reliable deployments.

Illustrations