How to Build Highly Available, Scalable, and Resilient Systems

Background

Reliable systems are the foundation for stable and fast‑growing businesses. Achieving high reliability and high availability requires a systematic approach.

High‑Availability Methodology

The following table (illustrated below) lists common high‑availability problems and their countermeasures.

Scalability

Scaling is the most common way to improve system reliability. It eliminates single points of failure and allows the system to handle increased traffic.

Scaling can be vertical (adding resources to a single node) or horizontal (adding more nodes). Vertical scaling improves capacity but does not remove single‑point failures; it is simple but limited. Horizontal scaling adds redundant nodes, provides strong capacity growth, but increases complexity and requires stateless, distributed design.

Scalability factor measures how much capacity increases when adding a unit of resources. Linear scalability means the factor stays constant.

Typical horizontal scaling patterns include deploying multiple application servers behind an Nginx load balancer with health checks, and using master‑slave database replication where the slave can take over if the master fails.

Isolation

Isolation limits resource consumption of individual services to prevent one service from exhausting the whole system. Isolation levels range from thread‑pool isolation, process isolation (e.g., Linux CGroup), module/application isolation, to data‑center isolation. Read‑write separation in databases is also a form of isolation.

Decoupling

Low coupling reduces maintenance cost. Decoupling can be achieved by interface‑based design, moving shared dependencies to a separate module, or replacing synchronous calls with asynchronous messaging (e.g., using Kafka, RabbitMQ, RocketMQ). Asynchronous messaging isolates failures and supports one‑way data flows.

Rate Limiting

Rate limiting protects a system from overload by controlling request volume. Common algorithms include Leaky Bucket, Token Bucket, and Sliding Window Counter. Rate limiting can be applied per‑instance (single‑machine) or globally, and can use fixed or dynamic thresholds based on load, CPU, or latency.

Leaky Bucket

The algorithm queues requests in a bucket that drains at a constant rate; excess requests are dropped. It can be implemented with a Redis queue where producers check the queue length before pushing messages.

Token Bucket

Tokens are added to a bucket at a steady rate; each request consumes a token. If no token is available, the request is rejected. The bucket’s capacity defines the allowed burst size. Guava’s RateLimiter uses this algorithm.

Sliding Window Counter

The method counts requests in recent time windows; if the count exceeds a threshold, traffic is limited. It relies on two parameters: window length and bucket interval.

Dynamic Rate Limiting

Instead of static thresholds, dynamic limiting adjusts limits based on real‑time metrics such as system load, CPU usage, or response latency.

Degradation

Business degradation sacrifices non‑core features to keep core functionality stable. It is typically controlled by feature flags in configuration systems (e.g., Alibaba’s Diamond, Ctrip’s Apollo, Baidu’s Disconf). Degraded services may show user‑friendly fallback messages.

Circuit Breaker

Inspired by electrical fuses, a circuit breaker prevents cascading failures by stopping calls to an unhealthy service. It has three states: Closed (normal), Open (calls short‑circuited), and Half‑Open (test calls to see if the service recovered). Netflix’s Hystrix is a popular implementation.

Release Practices

Automated module‑level testing isolates changes to a single module, reducing test scope and cost. Data can be collected via AOP or instrumentation points and replayed in an offline environment.

Gray‑scale releases gradually roll out changes (e.g., 1% → 10% → 100%) using representative user groups (canary). A clear rollback plan is essential for rapid recovery.

Fault Drills

Chaos engineering tools like Netflix’s Chaos Monkey intentionally induce failures to verify system resilience.

Automated Operations – Self‑Healing

Techniques such as hardware fault prediction, automatic server isolation, service self‑healing, and cluster rebalance can close the loop on hardware failures without human intervention.

Event Systems

Event logging services (e.g., AWS CloudTrail) record critical changes, enabling quick root‑cause analysis when incidents occur.

Other Design Considerations

Set reasonable timeouts for external calls to avoid blocking.

Implement retry policies carefully to balance success rate against increased load and potential cascading failures.

Summary

The article provides a comprehensive methodology for building high‑availability, scalable, and resilient systems, covering architectural patterns, isolation techniques, decoupling strategies, rate‑limiting algorithms, degradation and circuit‑breaker mechanisms, release engineering, fault‑injection testing, and automated self‑healing operations.