Why High Availability Isn’t About Preventing Failures but Surviving Them

Before studying high availability, the author mistakenly thought it meant keeping a system from ever failing; the article clarifies that high availability is about keeping services running after a failure.

01

High availability is not "no failure" but "still running after failure". A single server, a single database node, or a single gateway creates a single point of failure. The first principle of high‑availability architecture is to eliminate such single points.

02

The most basic technique is redundancy: deploy multiple web servers, multiple instances of an order service, master‑slave or primary‑backup databases, and multiple data‑centers for disaster recovery. Redundancy is not waste; it provides a fallback path for the system.

03

Redundancy alone is insufficient; you must detect failures. Mechanisms such as heartbeat checks, health checks, monitoring alerts, and service‑registry eviction identify unhealthy nodes. For example, if an order service has three instances (A, B, C) and A crashes, the system must detect A’s unhealthiness; otherwise the load balancer would keep sending requests to A, causing user failures.

04

After detecting a fault, you must perform failover: shift traffic or services to healthy nodes. When order service A goes down, the load balancer stops routing requests to A and redirects them to B and C. In databases, a standby can be promoted to primary when the master fails, allowing the application to continue operating.

05

Degradation is another crucial technique. Under heavy load or when dependent services are unavailable, non‑essential features can be temporarily disabled while preserving core functions such as browsing, ordering, and payment. This sacrifices secondary features (comments, recommendations, points, complex reports) to keep the business alive.

06

Master‑backup architecture: one primary node works while a backup node synchronizes data and stands by. When the primary fails, the backup takes over. Advantages are simple structure and suitability for disaster recovery; disadvantages include low backup utilization and possible brief pause during switchover.

07

Master‑slave architecture, common in databases: the master handles writes, the slave replicates data and typically handles reads. This improves read capacity and reduces load on the master, but replication lag can cause stale reads, so it is not inherently strongly consistent.

08

Cluster: multiple nodes jointly provide a service, usually behind a load balancer. Examples include three order‑service instances, five product‑service instances, multiple web servers, and multiple Redis nodes. Clustering improves processing capacity and ensures that if one node fails, others continue serving.

09

Active‑active (dual‑active) setup: two data‑centers (e.g., Beijing and Shanghai) serve users simultaneously. If one center fails, the other can take over part or all traffic. The difficulty lies in complex data synchronization, consistency handling, and network latency.

10

Multi‑active setup: several regions (North China, East China, South China) serve users concurrently, offering stronger disaster recovery and proximity latency benefits. The trade‑off is higher architectural complexity, harder data consistency, and increased operational cost.

11

RTO (Recovery Time Objective) defines the maximum allowed downtime after a failure, e.g., 10 minutes, meaning the system must be restored within that window.

12

RPO (Recovery Point Objective) defines the maximum data loss tolerated, e.g., 5 minutes, meaning after recovery the system may lose at most the last five minutes of data.

13

Sample exam answer for "How to improve high availability of an e‑commerce system?":

Deploy multiple instances of each service with load balancing to avoid single‑instance failures.

Run service‑registry and configuration‑center clusters to prevent them from being single points.

Use master‑slave or primary‑backup databases with automatic failover.

Cluster middleware such as Redis and Kafka to keep caching and messaging available.

Apply degradation: keep browsing, ordering, and payment functional while disabling non‑core features.

Implement monitoring, alerts, health checks, and automatic switch‑over to shorten RTO.

Perform data backup, log shipping, and cross‑region replication to reduce RPO.

The answer should combine technical measures, business objects, and their effects rather than merely stating "use clustering".

Self‑test

What problem does high availability solve? Answer: It reduces prolonged unavailability by using redundancy, fault detection, failover, and degradation to ensure business continuity.

What is a single point of failure? Answer: A component whose failure makes the entire system unavailable.

What does redundancy mean? Answer: Deploying multiple instances, nodes, or data‑centers for critical components.

What is master‑backup? Answer: The primary works while the backup stands by to take over on failure.

What is master‑slave? Answer: The master handles writes; the slave replicates data and typically handles reads.

Why does a cluster improve availability? Answer: Multiple nodes provide service together, and a failed node’s traffic can be rerouted by the load balancer.

How do dual‑active and master‑backup differ? Answer: Master‑backup has only the primary serving traffic; dual‑active has two centers serving traffic simultaneously.

What is RTO? Answer: The maximum allowed recovery time after a failure.

What is RPO? Answer: The maximum allowed data loss window after a failure.

What does degradation mean? Answer: Sacrificing non‑core functions to guarantee core business availability.

Key takeaways

If you remember only six sentences, they are:

High availability: reduce downtime and keep business continuous.

Redundancy: no critical component should be single.

Failover: switch traffic to a usable node after a failure.

Degradation: protect core functions, drop secondary ones.

RTO: how long downtime is acceptable.

RPO: how much data loss is acceptable.

High availability is not just "the system should be stable"; it is a concrete set of actions: prepare multiples, detect failures promptly, redirect traffic, optionally degrade, and minimize both downtime and data loss.