From Manual DNS to 90-Second Auto-Failover: Traffic Recovery Evolution at 10M QPS

What Traffic Recovery Actually Recovers

Data recovery answers whether data exists and is complete; traffic recovery answers whether requests can still enter and be correctly processed. In a database primary failure, data recovery promotes a replica and replays unsynced writes; traffic recovery moves tens of millions of requests from the old address to the new one without causing a secondary avalanche.

Traffic recovery operates at three layers with vastly different granularity and speed:

Node-level : A single service instance fails or slows; the load balancer removes it from the backend pool. Health checks run at second intervals; removal completes in seconds.

Data-center-level : Hundreds of services and thousands of machines become unreachable. The entire ingress surface must move: DNS re-resolution, scheduler recalculation, target data-center capacity confirmation. This is the true watershed — manual operation is inevitably slow.

Region-level : Rare, used for disaster drills or extreme disasters; shares the same underlying mechanisms as data-center-level but adds capacity and cost trade-offs.

Key insight : The first question is not "how to switch" but "which layer to switch". Mismatched layers cause either over-reaction (triggering data-center failover for a single instance blip) or under-reaction (waiting for node-level self-healing when the whole data-center network is down).

Traffic recovery and replica failover (covered in the previous lecture) often happen together but are independent: database primary/secondary switch is handled by the database HA component; business ingress traffic switch is handled by the traffic recovery system. Many switch accidents stem from misalignment between these two mechanisms.

Manual Switching: Where It Is Slow and Risky

Early traffic recovery was purely manual: ops engineers logged into consoles to modify DNS records or load-balancer upstream configs. The problem is not just slowness; every step depends on humans making correct decisions under pressure.

A typical chain breaks down as: alert confirmation (8 min), decision alignment (6 min), finding someone with DNS permission (4 min), config change + propagation (20+ min), repeated verification. Zero technical bottlenecks — all time consumed by "people finding people" and "people waiting for confirmation".

Five serial steps, each waiting on a human. The manual RTO floor is set by organizational response speed, not technology — an uncontrollable variable: the on-call may be handling another incident, the permission holder may have phone on silent, the decision-maker may hesitate over "what if we switch wrong".

Beyond slowness, three deeper dangers:

Uncontrollable DNS propagation : TTL is set to 60s but ISP LocalDNS often ignores it; actual propagation can take 5+ minutes. Real RTO is outside your control.

Operational risk : At 4 AM, a typo or wrong target data-center turns a small fault into a major one. Postmortems show "switch operation itself caused secondary fault" appears far more often than expected.

Playbook rot : Manual switching relies on runbooks that expire — domain configs change, permission holders leave, console paths disappear. A playbook executed once a year has real reliability far below its documented promise.

Conclusion : Manual switching can be a starting point, but its ceiling is visible. As long as humans execute, RTO cannot break into sub-minute territory.

Semi-Automatic: Human Still in the Loop

The natural next step: script the operations. Switching actions become pre-written, tested scripts; humans only pull the trigger. This eliminates most operational risk — scripts are pre-validated, idempotent, include rollback steps. Switching changes from "live improvisation" to "executing rehearsed actions".

But semi-automatic retains two human judgments: fault confirmation and switch decision .

Fault confirmation needs humans because single-source alerts have non-zero false-positive rates at scale: a network blip or monitoring component failure can emit "data-center down" alerts.

Switch decision needs humans because data-center failover impacts all same-city businesses and requires cross-team coordination — an inherently organizational decision.

Most expensive error in traffic recovery is not switching too slow, but switching wrong : diverting traffic from a healthy data-center equals manufacturing a fault. Letting an automated system directly react to high-severity "data-center" alerts amplifies the cost of false positives to the maximum.

Typical semi-automatic form: a switch platform that codifies actions, checks, rollbacks into a "playbook ticket". On-call opens ticket, confirms key metrics, clicks execute; platform runs and outputs verification report. Switch time drops from 30 min to under 5 min — major progress.

Limitation: bottleneck shifts from "human operation speed" to "human judgment speed". Judgment still requires the human to be awake, online, and decisive. At 3 AM with a large blast radius and wavering metrics, the "to switch or not" decision can take minutes — the minutes when business bleeds fastest.

Structural issue: playbooks cover only "anticipated faults". Real faults are always combination punches. When actual fault deviates from playbook assumptions, the executor falls back to manual mode, reconciling script vs reality — often slower.

Fully Automatic: Handing Judgment to the System

Full automation must solve both confirmation and decision. Without this step, RTO stays at minutes; with it, seconds become possible.

Premise: the system must judge more accurately than humans. Three core problems must be solved:

1. Fault Determination Must Be Accurate

Single-point probing is unreliable. Mature approach: multi-source cross-voting — multiple independent probes (different network paths, different methods) simultaneously judge unreachability; majority consensus required. This applies Paxos-like quorum thinking to the detection layer.

Auto-switch determination logic prefers a few seconds of delay over triggering a high-risk action on a single signal. A few seconds of determination latency is trivial; a false switch is an incident. Engineering typically adds an "observation window": fault signal must persist beyond a time window (e.g., 30s) before entering decision, specifically to filter transient jitter.

2. Target Capacity Must Be Stable

The receiving data-center must handle the load. Counter-intuitive point: failure moments often coincide with abnormal traffic; the target may show headroom normally, but the instant traffic shape (retry storms, cache penetration) differs completely. Therefore auto-switch must perform capacity verification before execution: confirm target water-level, error-rate, dependency health are all green; otherwise defer rather than force-cut.

"Can switch" and "switch without blowing up" are two different things; the auto-switch system must answer the latter. Common practice: reserve fixed capacity headroom per data-center (e.g., daily load ≤ 50%), design playbooks against that headroom, and make "capacity water-level" part of daily capacity management — not calculated on the fly during a switch.

3. Execution Rhythm Must Be Controllable

Cutting tens of millions of QPS in one shot saturates target caches, connection pools, thread pools instantly, turning "partial fault" into "global avalanche". Hence auto-switch is batched : cut 5% → observe → 20% → 50%, each batch gated by health checks. Overall switch takes slightly longer, but every batch is rollback-capable; total risk is lower.

Note the rollback loop in the diagram. Automatic rollback capability is not insurance — it is the prerequisite that makes "automatic" viable.

Auto-switch without auto-rollback merely swaps human-error risk for system-misjudgment risk; total risk does not decrease.

With these three solved, switching becomes a true mechanism: no more "pulling a group chat"; detection → determination → execution becomes a pipeline, yielding 90-second data-center failover.

Three-Layer Traffic Scheduling Implementation

Auto-switch is the goal; execution relies on concrete scheduling facilities. Traffic from user to service instance passes through several layers, each capable of switching, with different speed and granularity. Understanding these three layers explains why large shops use all three in concert rather than picking one.

Layer 1: DNS & Global Traffic Scheduling (GSLB)

User resolves domain → which data-center ingress IP → traffic goes there. Traditional DNS switch modifies records; slow, uncontrollable. Modern GSLB: routes users by geography/ISP to nearest ingress, retains "one-click redirect all resolutions for a region to another data-center" capability, compresses TTL to sub-minute. DNS layer suits data-center/region coarse-grained switch; advantage: broad coverage; disadvantage: minute-level propagation latency, helpless against non-TTL-compliant clients.

Layer 2: Load Balancer Layer

Traffic reaches data-center ingress; L4/L7 load balancer distributes to backends. This layer switches in real-time: change an upstream config, remove a backend group, seconds to effect. It handles fine-grained distribution after DNS-layer cut, and can independently perform intra-data-center switching. Key: reliability of config push channel — syncing config to thousands of devices in seconds is itself a mini distributed system.

Layer 3: Client-Side & Service Discovery

In microservices, callers fetch instance lists from registry and do client-side LB. Finest granularity (per-service, per-group), fastest effect (push = immediate), but coverage limited to internal calls using this stack.

Three-layer summary : DNS decides which door users enter; load balancer decides which machine after the door; service discovery decides who internal calls find. Data-center failover typically orchestrates all three: DNS moves ingress first, load balancer absorbs, service discovery follows.

Single-layer systems have clear gaps: DNS-only leaves internal calls looping in failed data-center; service-discovery-only leaves user ingress pouring into failed data-center; load-balancer-only has no healthy upstreams during data-center failure. Three-layer coordination seems complex but reflects a fact: switching capability must span the full traffic path.

Real-world constraint: CDN. Static-asset businesses have CDN origin-pull as part of the traffic path. If CDN origin still points to failed data-center during switch, static assets break. Mature switch systems fold CDN origin switch into the same playbook, typically sequenced after ingress switch.

Preventing Auto-Switch from Becoming a New Fault Source

Once auto-switch capability exists, a new risk emerges: the switch system itself becomes a fault source. Not theoretical — industry has famous incidents where "automation misjudgment triggered massive switch". The more powerful the automation, the higher the cost of its misfire; protection mechanisms must scale with automation level.

Switch thresholds : Multi-source voting + observation window (first-class thresholds). Stricter class: high-risk switch requires multiple independent conditions simultaneously — e.g., "local probes fail" AND "peer data-center healthy" AND "peer capacity sufficient" AND "no human freeze in last N minutes". Conditions are ANDed; any single probe anomaly cannot trigger switch alone.

Switch locks : Switching is stateful — a system mid-switch must reject a second switch. Global locks and object locks enforce this. Hidden lock: during change-freeze windows (major promotions, big releases), auto-switch degrades to alert-only . System state is in flux, signal credibility drops; automation yields to human judgment.

Anti-flapping : Post-switch, if determination signals oscillate near boundary, system may thrash "cut over → cut back". Engineering fix: cooldown period after switch — freeze same-direction determination, or raise return threshold (exit at 50% votes, return requires 70%), using asymmetric conditions to break oscillation.

Human-machine boundary : Full auto ≠ no human. Precise statement: auto-system executes low-risk, deterministic actions; high-risk, ambiguous actions auto-system only prepares and escalates for one-click human confirmation. Node-level removal, small-percentage cuts fully auto; data-center full cut auto-prepares everything, pops up for human confirm. Boundary placement varies by org but always exists — the balance point between automation and accountability.

These protections sound like pouring cold water on automation, but they are exactly why automation survives long-term. A system that suffers one major misjudgment gets permanently "turned off" or "downgraded to alert"; prior investment wasted. Protection mechanisms are not automation's cost — they are its insurance.

Switch Capability Is Honed Through Drills

Like data recovery, traffic switch capability's true quality is only known during real execution. Every number in the playbook — execution time, target capacity, rollback time — is an estimate until backed by drill data.

Drills typically run at three levels:

Component-level : Verify single mechanisms — e.g., how long for LB to converge after node removal; registry push latency.

Data-center-level : Cut entire data-center ingress traffic; observe if target absorbs as expected; error-rate and latency curves.

Region-level / full-chain : Simulate extreme fault surfaces; test multi-system coordination.

At 10M QPS scale, data-center drills run multiple times per year; some teams achieve "routine network-partition drills": periodically physically disconnect a data-center network, watch system auto-recover. Value extends beyond verifying the switch — exposes hidden issues: services with hard data-center dependencies, caches lacking pre-warm, monitoring losing coverage post-switch.

80% of issues exposed by switch drills are not switch-system issues; they are business systems that assumed the data-center would always exist.

Hidden drill dividend: turns "switch" from a heavily-approved major event into a routine ops task. When an action runs 10 times a year and succeeds every time, organizational trust builds; the "human confirm" line in the human-machine boundary can finally be retracted. Trust is not built on promises — it is built on repeatable success records.

From Emergency Action to Routine Capability

Viewing the three phases together, the evolution path is clear:

The table's key column is not the duration numbers but the risk column's transformation: risk does not disappear, it changes form. Manual era risk = human operational uncertainty; auto era risk = system misjudgment; protection mechanisms are the solution to this new risk.

Traffic recovery evolution is not a process of eliminating risk, but of converting risk from uncontrollable human factors into designable, verifiable system mechanisms.

Overlooked perspective: building traffic recovery capability reshapes architecture itself. When data-center failover becomes a sub-minute routine operation, dual-active or multi-active architectures dare carry real traffic instead of sitting idle; "multi-active" shifts from DR concept to capacity strategy; traffic can be freely scheduled between data-centers following capacity water-levels. At this point switch capability is no longer just a fault lifeline — it becomes a daily capacity management lever. Many companies' multi-active architectures were not planned upfront; they grew naturally after switch capability matured.

Back to the opening 47-minute switch. Postmortem conclusion in one sentence: codify human judgment into system determination rules, codify human operations into system execution actions, then feed every execution's data back to refine determination rules. No shortcuts on this path, but every step has definite ROI. If your traffic switch still lives in document playbooks, the first worthwhile action is not writing automation — it is running a real cutover drill to capture your first measured RTO. When did you last let traffic truly leave your primary data-center?