Understanding High Availability and High Performance: Complexity, Redundancy, and Decision Strategies

We begin by defining high availability (HA) as the ability of a system to continuously perform its functions without interruption, noting that true "no‑downtime" is impossible due to hardware failures, software bugs, aging components, and external disasters such as power outages or earthquakes.

To achieve HA, redundancy is introduced—adding more machines or data‑centers to eliminate single points of failure. While redundancy improves availability, it also increases system complexity, requiring careful analysis of each scenario to select appropriate HA solutions.

Compute HA

For computational tasks, HA means that the same algorithm and input should produce identical results on any machine, allowing seamless migration of workloads. Moving from a single‑node to a dual‑node architecture introduces new complexities such as task schedulers, connection management, and allocation algorithms (e.g., active‑standby, active‑active, cold/warm/hot standby).

More complex HA clusters further increase the number of possible master‑backup configurations (e.g., 1‑master‑3‑backup, 2‑master‑2‑backup) and require careful algorithm selection based on business needs.

Storage HA

Storage HA faces additional challenges because data must be transferred across network links, introducing latency (milliseconds within a data‑center, tens to hundreds of milliseconds across regions). This latency inevitably leads to temporary data inconsistency, which can cause serious business issues such as incorrect account balances.

Network failures (cable cuts, congestion, packet loss) can further prolong unavailability, as illustrated by real‑world incidents like the 2015 Alipay outage caused by a cut fiber cable.

The CAP theorem formalizes the trade‑off: a storage system cannot simultaneously guarantee consistency, availability, and partition tolerance; designers must choose two based on business priorities.

HA State Decision

Effective HA requires accurate state decision—determining whether the system is normal or abnormal. Three decision models are discussed:

Dictatorial : a single decision maker collects status from replicas; if the leader fails, the whole decision process collapses.

Negotiation : two nodes exchange status and elect a master; failures in communication can cause split‑brain or loss of a master.

Democratic : multiple nodes vote (e.g., ZooKeeper’s leader election); while robust, it can suffer from split‑brain scenarios unless a majority quorum is enforced.

Each model introduces its own complexity and cannot guarantee flawless operation in every scenario.

High Performance Complexity

High performance also adds complexity at two levels: inside a single machine (processes, threads, scheduling, SMP/NUMA/MPP) and across clusters (load balancing, sharding, replication, parallelism).

Single‑Machine Complexity

Operating systems manage processes and threads to keep CPUs busy, evolving from batch processing to multi‑process, multi‑threaded, and multi‑core architectures (SMP). Choosing the right concurrency model (e.g., Nginx’s multi‑process vs. multi‑thread) requires deep analysis of workload characteristics.

Cluster Complexity

Scaling beyond a single machine demands task distribution, data partitioning, replication, and parallel execution (e.g., MapReduce). Adding more machines introduces new components such as load balancers, DNS‑based routing, and multi‑layer task dispatchers, each adding configuration and failure‑handling overhead.

When request rates reach 100 000 RPS, the task dispatcher itself becomes a bottleneck, requiring multiple dispatchers, DNS round‑robin, GSLB, or CDN solutions, and a many‑to‑many connection topology between dispatchers and business servers.

Task Decomposition

Beyond simple task allocation, breaking a monolithic service into smaller sub‑systems (e.g., registration, messaging, LBS) allows targeted scaling and optimization. However, excessive decomposition increases inter‑service network calls, which can degrade performance; a balance must be struck.

In summary, achieving high availability and high performance inevitably raises system complexity; designers must weigh redundancy, consistency, decision mechanisms, and granularity of decomposition to meet reliability and speed goals.