17-year internet software developer specializing in AI applications, networking, architecture, and open source. Led the delivery of network services handling hundreds of millions of concurrent devices and tens of millions of QPS, and has three years of experience designing and building an agent platform. Follow to stay updated.
This article details the six-stage evolution of fault postmortems—from blameless culture and data-driven timelines to structured root cause analysis, prioritized improvement loops, and automated platforms—showing how 10M QPS systems transform incidents into reusable systemic assets rather than blame exercises.
At 10M QPS, serial restarts of 4,200 instances take 47 hours while naive parallel restarts cause cascading failures; this article details the evolution from graceful single-node shutdown and warm startup to batched parallel orchestration with health gates, restart-storm mitigation, stateful-service handling, and adaptive platform automation.
This article details the evolution of cache reconstruction strategies for ten-million-QPS systems, covering cold-start dangers, persistence recovery limits, tiered hot-data backfill, dual-cluster hot standby for second-level failover, shard-level background rebuild with singleflight, and multi-layer database protection during rebuilds, advocating making pre-warming a routine capability.
This article traces the evolution of traffic recovery from manual DNS changes taking 47 minutes to fully automated 90-second failover, detailing the three-layer scheduling architecture, multi-source voting, capacity verification, batched switching, and protection mechanisms that make automatic failover safe at 10M QPS scale.
This article traces the evolution of data recovery from manual backup restoration through log replay to automated replica switching, explaining RPO/RTO trade-offs, and details how 10M QPS systems implement shard-level automatic rebuild with throttling, verification, and continuous recovery as a runtime capability.
This article decomposes recovery time into detection, decision, execution, and verification phases, showing how 10M QPS systems reduce recovery from hours to minutes by pre-positioning decisions at design time, bounding data loss via semi-sync replication, automating failover loops with fencing and quorum, and validating RTO through disciplined drills.
This article dissects why static DNS-based traffic scheduling fails at ten-million-QPS scale and how layered dynamic scheduling — across access, gateway, service, and data layers — combined with unitized routing, automated decision loops, and rigorous drill practices enables precise, reversible, and safe failover.
This article outlines a five-level maturity model for disaster recovery drills—tabletop exercises, component drills, normalized chaos engineering, full-link switchover, and calendar-based routines—to transform static plans into validated capabilities, measuring real RTO/RPO and blast radius for 10M QPS systems.
The article presents a layered fault-domain strategy—physical anti-affinity, logical isolation (sharding, cluster groups, swimlanes, bulkheads), cell-based architecture, chaos-engineering validation, and quantitative governance metrics—to shrink the blast radius of a ten-million-QPS system from a fixed 100% to a controllable 1/N design parameter.
This article systematically dissects the engineering evolution from disaster backup to multi-active architecture, covering fault domain isolation, RTO/RPO/recovery capacity metrics, data replication trade-offs, traffic switching state machines, business unitization, conflict convergence, and a phased adoption roadmap for systems operating at ten million QPS.
