Designing High‑Availability Systems: Architecture, Capacity Planning, and Fault‑Tolerance Guide

Background and Availability Metrics

System availability is expressed as a percentage of normal operation time, commonly measured in "nines" (e.g., 99.99% = four nines). The availability formula is Availability = (1 - FaultTime / TotalTime) × 100%, where fault time equals the interval between fault discovery and fault resolution.

Fault Prevention

Application‑level prevention: load balancing, auto‑scaling, asynchronous decoupling, fault tolerance, overload protection.

Data‑level prevention: redundant backups (hot/cold), failover mechanisms.

Change‑management prevention: standardized module change processes, approval workflows for core modules, new data sets, model releases, and SDK updates.

Health‑check system: real‑time collection of module metrics (resource usage, process status, service health) to quickly identify abnormal modules.

Fault Detection

Release testing, monitoring alerts, disaster recovery drills, and fault‑injection exercises.

Four‑star alarm cycle governance to improve accuracy, reliability, and timeliness of alerts.

Fault‑dashboard for a holistic view of service health.

Fault Recovery

Incident response plan: define clear recovery procedures for each fault scenario.

Cut‑over tools: enable traffic migration to other availability zones or data centers when a region fails.

Rapid rollback: support immediate rollback to a stable version.

Capacity Evaluation and Planning

Estimate QPS (queries per second) based on product and operational input for new systems, or use historical data for existing services. Capacity planning should consider overall system load, then break down to individual micro‑service requirements. The process includes:

Collect business‑level QPS estimates (average and peak).

Validate estimates with historical metrics.

Design system architecture to meet the projected load.

Performance Stress Testing

Conduct full‑stack load tests to verify that the system can sustain the target QPS and response latency. The goals are to confirm capacity planning accuracy and expose performance bottlenecks.

Layered Architecture Design

The system is divided into four layers:

Access Layer: entry points, DNS security, DDoS protection, and API gateway for routing, rate‑limiting, and security enforcement.

Application Layer: stateless services that present functionality, support horizontal scaling, and enable gray‑release and A/B testing.

Service Layer: domain‑specific micro‑services, each isolated and independently deployable.

Data Layer: databases and storage, with redundancy and failover to guarantee data durability.

Service Tiering and Governance

Services are classified into four tiers based on criticality and required availability:

Tier 1 – Core Services: 99.99% availability (≈53 minutes downtime per year). Deployed with N+1 redundancy, full monitoring, rapid rollback, and strict isolation.

Tier 2 – Important Services: 99.95% availability (≈260 minutes downtime per year). Similar redundancy and monitoring as Tier 1 but may share servers.

Tier 3 – General Services: 99.9% availability (≈8.8 hours downtime per year). Single‑point deployment is acceptable; monitoring focuses on process health.

Tier 4 – Tooling Services: 99.9% availability but with minimal operational requirements; often deployed on shared infrastructure.

Resilience Mechanisms

Failover: automatic switch to a standby instance when the primary fails (N+1 deployment).

Failfast: immediate error return on first failure to avoid cascading delays.

Failback: automatic restoration of the primary after the fault is resolved.

Failsafe: ignore non‑critical errors to prevent system‑wide impact (e.g., logging failures).

Idempotent Design: ensure repeated calls produce the same result, crucial for retry and failover scenarios.

Circuit Breaker (e.g., Resilience4j): fast‑fail, rate‑limit, and isolate faulty downstream services.

Operational Practices

Standardization: unified coding standards, repository structures, and CI/CD pipelines.

Automation: CI/CD for automated builds, tests, and deployments; support gray‑release and canary strategies.

Monitoring & Alerting: multi‑layer observability (network, system, application, business metrics) with automated remediation (e.g., auto‑restart on GC spikes).

Full‑stack Observability: trace, log aggregation, and metric collection to reduce MTTR.

Fault Drills: regular chaos engineering and disaster‑recovery exercises to validate recovery procedures.

Roles and Responsibilities

Architect: design high‑availability architecture, coordinate with operations, define standards, and ensure system scalability and reliability.

Operations / SRE: maintain monitoring, automate deployments, manage disaster recovery, and enforce operational standards.

Developers: implement designs following coding standards, perform unit and integration testing, and support automated pipelines.

Clear responsibility boundaries prevent blame‑shifting and accelerate issue resolution, ultimately improving overall service availability.