The article discusses common monitoring challenges such as fragmented tooling and noisy alerts, and proposes solutions including consolidating to a single monitoring framework, prioritizing runtime exceptions, and classifying business alerts with codes and trace information to improve incident response.
This article presents a systematic, data‑model‑driven approach to Kubernetes stability assurance, detailing the sources of complexity, a four‑diagram and three‑table data model, insight and pre‑plan structures, global visualisation concepts, deployment patterns, operational workflows, and competitive analysis to enable effective, iterative, and sustainable cluster stability management.
This guide explains why monitoring is essential throughout a product lifecycle, outlines monitoring modes and methods, compares health checks, logs, tracing and metric solutions, and provides a detailed Prometheus‑based monitoring architecture with concrete metric definitions, alerting rules, and incident‑response procedures.
A routine release introduced a tiny code change that triggered a massive production outage, causing millions in losses; the team’s swift rollback, post‑mortem analysis, and reflections on code discipline, testing, and process compliance highlight essential lessons for reliable backend operations.
A seemingly harmless code change that returned null triggered a massive production outage, costing millions, and the author recounts the incident, the emergency rollback, root‑cause analysis, and the broader lessons about code review, testing, monitoring, and disciplined release practices.
This article shares practical insights from a technical leader on designing robust system architecture, implementing comprehensive capacity planning, establishing reliable operations processes, strengthening security, and cultivating team awareness to achieve long‑term stability for large‑scale internet services.
This article walks through a real call‑center outage, outlines why traditional ad‑hoc debugging fails, and presents a structured approach—including symptom identification, rapid root‑cause isolation, enhanced monitoring, concise emergency playbooks, and intelligent automation—to dramatically reduce recovery time and move toward self‑healing operations.
This article details a real‑world MySQL production data loss incident, explains how to identify the relevant binlog range, use binlog2sql and MyFlash to generate reverse SQL or binary logs, and outlines the step‑by‑step recovery process and post‑mortem reflections for operations teams.
This guide walks through a real‑world call‑center slowdown incident, outlines common fault‑handling techniques, proposes monitoring enhancements, details a comprehensive emergency‑response plan, and introduces intelligent event‑processing concepts to help operations teams resolve outages faster and more reliably.
This guide outlines essential incident‑response principles—prioritizing business restoration and timely escalation—while detailing practical methods such as restart, isolation, and degradation, and explains how to organize response teams and conduct thorough post‑incident reviews.
This guide outlines essential incident management principles—prioritizing business restoration and timely escalation—followed by detailed methodologies such as restart, isolation, and degradation, and explains role responsibilities, user impact handling, and post‑incident summarization for continuous improvement.
The article outlines a three‑stage ops maturity model—firefighting, fire prevention, and arson—explains how proactive fault‑injection drills, continuous availability improvements, and aligning technical metrics with business value can transform operations from reactive responders into strategic value creators.
Drawing on five years of incident reviews, the article defines a comprehensive stability framework for backend services—mandating timeout hierarchies, weak dependencies, service-discovery integration, staged gray releases, robust monitoring, capacity planning, and strict change management—while cataloguing common anti-patterns such as over-aggressive circuit breaking, static retries, improper timeouts, tight coupling, and insufficient isolation, and urging regular rehearsal of these practices.
Incident Managementbackend stabilitydeployment best practices
This guide explains fault management fundamentals—from ITIL‑based definitions and why it matters, to fault level classification, monitoring, emergency response, recovery, post‑mortem analysis, continuous improvement, and practical advice for practitioners—providing a comprehensive, actionable framework for reliable operations.
The article presents a detailed Google‑derived post‑mortem checklist covering event data collection, root‑cause analysis, lessons learned, actionable improvement items, and review procedures to ensure systematic, non‑blame‑focused incident handling.
Incident ManagementRoot Cause Analysisaction items
This article explains how focusing on three key questions during incident post‑mortems, balancing business speed with system stability, and establishing clear SLOs can turn failures into actionable improvements and better fault‑tolerance strategies.
The article details 360's AIOps deployment, covering external speaker insights, internal architecture, data collection pipelines, AI models for resource recycling, alarm reduction, and correlation, as well as visualization dashboards, labeling platforms, and self‑healing mechanisms, illustrating a comprehensive AI‑driven operations framework.
The Open Alarm Platform provides a flexible data model, modular architecture, and robust stability features to enable various business lines to integrate their custom monitoring systems via APIs, offering alert convergence, merging, multi‑channel delivery, and comprehensive management while reducing development and maintenance costs.
The article reflects on handling online incidents by first extinguishing the immediate problem, then digging into root causes, and expands the discussion to what truly constitutes technical ability, the pitfalls of reinventing solutions, raising one’s perspective, and the critical role of systematic retrospection.
The article draws parallels between emergency rooms and IT operations, describing how Kanban's WIP limits, one‑to‑one liaison models, transparent dashboards, and Scrum time‑boxing (daily stand‑ups and weekly reviews) help a globally distributed team prioritize urgent incidents while still advancing important non‑urgent work.
A collection of vivid DBA anecdotes reveals common pitfalls—from missed alerts and accidental production restarts to unsafe terminal habits—and distills practical safeguards that any database operator can adopt to avoid costly mishaps.
A personal account of switching jobs and the first day on an operations team, detailing a critical menu‑click error, the rapid response, and the deeper reflections on risk awareness, teamwork, and professional growth in high‑stakes production environments.
This article outlines Youzan's end‑to‑end online incident management process—from fault detection and coordination through root‑cause analysis, recovery, review, and actionable JIRA tracking—highlighting practical workflows, data analysis, and continuous improvement practices for reliable service delivery.
This talk explores how ops engineers can achieve continuous professional development by adopting checklist thinking, covering growth drivers, error classification, practical checklist applications, cognitive models, and design principles that turn complex incidents into systematic, repeatable processes.
This article examines the growing challenges of system availability in modern operations, explains the concept of availability and the N‑nine metric, introduces Business Continuity Management and Google SRE approaches, and provides concrete technical and managerial methods—including architecture standardization, scaling strategies, tooling, emergency drills, and incident‑centralized management—to improve operational reliability.
A personal account of switching jobs, the challenges of a first production shift, a critical menu‑click error, the team’s rapid response, and the lasting operational lessons learned about risk awareness, double‑checking, and continuous improvement.
This article shares practical SRE‑based principles for diagnosing and resolving online incidents, emphasizing systematic investigation, gathering clues, and prioritizing service restoration over immediate root‑cause identification to make troubleshooting less mystical and more effective.
The article describes how a ticketing system with strict SLA enforcement, automated routing, and team‑based service management enabled rapid, orderly issue handling during the high‑volume Double 11 shopping event, achieving near‑90% resolution within 30 minutes and improving overall business stability.
This article outlines a systematic methodology for understanding, evaluating, and quickly resolving production system incidents by categorizing system layers, assessing impact, employing Linux diagnostic tools, and designing fault‑tolerant mechanisms to minimize downtime and maintain core functionality.
This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.
This reading note introduces the DevOps three‑step workflow from The Phoenix Project, outlines its three core practices, and summarizes fifteen scenarios illustrating common operational challenges such as untested deployments, missing change records, incident response, and improvement actions.
This article explains how Tencent's social platform operations team prepares for massive holiday traffic spikes by following a four‑stage process—business preparation, capacity evaluation, resource provisioning, and scaling with stress testing—while detailing team structures, operational standards, and the supporting tool ecosystem that enable reliable, high‑availability services.
A sudden rain‑triggered alert cascade exposed a mis‑tested data‑migration tool that overwrote production routing, causing a DB success‑rate drop to zero; the incident narrative details detection, root‑cause tracing, and a set of DevOps tool‑development standards to prevent future outages.
This article shares practical SRE‑based principles and step‑by‑step methods for diagnosing and resolving online incidents, emphasizing mindset, systematic information gathering, and structured analysis to turn mysterious outages into solvable problems.
This article shares practical insights from Google SRE on on‑call duty, covering why on‑call is needed, common challenges, effective scheduling, evaluation methods, and actionable tips to improve team resilience and reduce stress for operations engineers.
A cautionary tale of a rookie who erased a production database on day one, followed by real‑world anecdotes, Reddit and Twitter reactions, and practical advice on backups, ticket handling, and strict operational rules to prevent costly data‑loss incidents.
This article outlines a comprehensive methodology for handling large‑scale online service incidents, covering goals, the "jump‑fill‑avoid" framework, step‑by‑step processes for detection, diagnosis, remediation, and post‑mortem analysis, as well as essential monitoring, logging, and escalation infrastructure.
When a Saturday early-morning outage left the company’s website and mobile app inaccessible for many users, the team traced the issue to an unpaid domain causing DNS resolution failures, detailed the investigation steps, temporary fixes, and lessons learned about DNS processes and operational readiness.
This article shares a detailed case study of how a financial services team restructured European application operations, applied lean retrospectives, built a top‑down monitoring system, and introduced systematic stakeholder collaboration to dramatically improve incident response, system robustness, and user satisfaction.
This article reviews sixteen notable 2016 service outages across finance, cloud, and entertainment, analyzes their causes—ranging from power failures to DDoS attacks—and highlights the critical need for robust disaster‑recovery and information‑security practices.
The article explores personal heroism in operations, defining it as reliance on individual effort to keep flawed systems appearing normal, examines its short‑term benefits and long‑term drawbacks for companies, teams, and the heroes themselves, and offers practical strategies to eliminate this risky mindset.
This article explains Google’s SRE postmortem philosophy, the criteria for writing postmortems, best practices for a blame‑free culture, and how collaborative knowledge‑sharing and incentives improve incident handling and overall system reliability.
This talk details Ctrip's application operations framework, covering data‑center scale, multi‑application deployment on Windows, high availability goals, capacity‑prediction models, disaster‑recovery design, incident response, and the evolution from manual tooling to automated, intelligent operations.
This article recounts two real‑world operations incidents—a network outage caused by an improperly configured portfast on a trunk link and an NFS failure that crippled an API service—then distills practical lessons on pre‑incident procedures, monitoring, fault handling, recovery, and post‑mortem practices.
This article examines recent high‑profile internet outages caused by human error, explores why operations teams are especially prone to mistakes despite automation and standards, and offers practical strategies—such as hiring the right people, fostering safety awareness, and turning professionalism into habit—to reduce future incidents.
This article summarizes a detailed Q&A from the Efficient Operations talk, covering BlueKing’s integration with databases, agent resource management, alarm de‑duplication, automation workflows, development language choices, data handling, and the platform’s suitability for various enterprise environments.
This article explores Randy Shoup’s interview on Google’s DevOps culture, detailing how high‑efficiency organizations instantly detect problems, use swarming to resolve them, document lessons as new knowledge, and foster a blameless post‑mortem culture that drives continuous improvement.
The article examines Ctrip’s prolonged service restoration after a May 28 incident, analyzing the complexities of SOA‑based architectures, the pitfalls of black‑box operations, and how transitioning to white‑box, DevOps‑aligned practices can prevent similar outages.