This article explains core fault‑tolerance principles for distributed systems, covering isolation, redundancy, health checks, failure detection, automatic recovery, consistency trade‑offs, Saga transactions, monitoring, prediction, and team practices to create resilient, maintainable architectures.
Bilibili’s infrastructure team created a lightweight, multi‑layered disaster‑recovery drill platform—combining an atomic fault library, scenario catalogs, chaos‑experiment orchestration, real‑time observation, and a product‑level interface—backed by standardized governance and CI‑integrated automation, cutting drill preparation from weeks to days and boosting weekly resilience testing across the organization.
This article details ByteDance’s disaster‑recovery evolution—from a single‑room deployment to same‑city multi‑data‑center setups and finally to active‑active multi‑region architectures—explaining the challenges, specific failure scenarios, and the strategic practices used to ensure continuous service during outages.
Drawing on twenty years of Google’s SRE experience, this article shares eleven practical lessons—from proportional incident mitigation and pre‑tested recovery mechanisms to canary releases, disaster‑resilience testing, and frequent deployments—aimed at improving reliability and operational efficiency.
Drawing on twenty years of Google SRE experience, this article outlines eleven practical lessons—from scaling mitigation to disaster‑resilience testing—that help teams design, operate, and evolve reliable large‑scale services.
The deck outlines post‑incident stability enhancements and the adoption of Service Level Objectives after the “713” fault, detailing failure analysis, reliability upgrades, monitoring practices, and the definition and operation of SLOs to sustain system quality, illustrated through architecture diagrams and reliability metrics.
Chaos engineering deliberately injects failures into distributed systems to measure resilience, using scientific experimentation to uncover hidden weaknesses, guide robust design, and improve reliability across development, testing, and production environments.
Chaos engineering, a discipline for experimenting on distributed systems, helps teams identify hidden weaknesses, improve high‑availability, and build confidence in production by defining stable states, injecting realistic failures, and measuring impact through observability metrics, with practical steps, tool choices, maturity stages, and evaluation methods.
Wireless operations and maintenance (O&M) evolved from backend‑focused practices to address stability and performance of mobile‑device services, tackling low issue detection rates and delayed responses through improved monitoring, gray‑release tagging, phased rollouts, AI‑driven diagnostics, and automated release gates, while inviting collaborative development.
This article explores Qunar's practical implementation of chaos engineering, detailing its value, the four strategic directions, shutdown and application drills, strong‑weak dependency handling, container support, and automated closed‑loop testing that together boost system resilience, process robustness, and user experience.
The article outlines a three‑stage ops maturity model—firefighting, fire prevention, and arson—explains how proactive fault‑injection drills, continuous availability improvements, and aligning technical metrics with business value can transform operations from reactive responders into strategic value creators.
This article describes Ctrip's SRE team's journey in adopting chaos engineering, outlining the motivations, roadmap, concrete experiments, platform maturity, and future automation goals to improve system resilience and operational reliability in a large‑scale microservice environment.
A seasoned operations professional argues that despite sensational claims, the ops function remains essential—driven by its core responsibilities of quality, cost, efficiency, and security, evolving with cloud computing, DevOps, and emerging IoT demands.
This article outlines what large‑scale website operations entail, describes the full product lifecycle involvement of ops engineers, lists the technical skills and personal qualities required, examines current industry issues, and highlights key technologies such as cluster management, monitoring, fault handling, and automation.
This talk walks through the transition from classic application operations to a DevOps culture, highlighting common pain points, the need for standardization and automation, and practical steps for engineers to evolve their skills and boost organizational efficiency.
This article introduces the standout operations professionals featured by the High‑Efficiency Operations community in 2016, summarizing each expert’s background, key achievements, and a curated list of their most influential technical articles for readers seeking deep insights into modern ops practices.
This talk details Ctrip's application operations framework, covering data‑center scale, multi‑application deployment on Windows, high availability goals, capacity‑prediction models, disaster‑recovery design, incident response, and the evolution from manual tooling to automated, intelligent operations.
A seasoned operations leader discusses how e‑commerce operational support differs from general internet applications, covering longer support chains, consistency models, seasonal traffic spikes, team role separation, mobile‑internet challenges, future planning, and the rise of enterprise‑level ops services.
