BestHub
Sign in
Tagged articles

incident management

146 articles · Page 2 of 2
macrozheng
macrozheng
Apr 24, 2021 · Operations

How a Single Code Change Caused Million-Dollar Loss and What It Taught Me About Release Discipline

A routine release introduced a tiny code change that triggered a massive production outage, causing millions in losses; the team’s swift rollback, post‑mortem analysis, and reflections on code discipline, testing, and process compliance highlight essential lessons for reliable backend operations.

code qualityincident managementrelease process
0 likes · 9 min read
How a Single Code Change Caused Million-Dollar Loss and What It Taught Me About Release Discipline
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 27, 2021 · Operations

How to Build Sustainable System Stability: Architecture, Ops, and Team Practices

This article shares practical insights from a technical leader on designing robust system architecture, implementing comprehensive capacity planning, establishing reliable operations processes, strengthening security, and cultivating team awareness to achieve long‑term stability for large‑scale internet services.

Capacity PlanningSoftware Engineeringarchitecture-design
0 likes · 24 min read
How to Build Sustainable System Stability: Architecture, Ops, and Team Practices
MaGe Linux Operations
MaGe Linux Operations
Jan 24, 2021 · Operations

How to Speed Up Call Center Incident Resolution with Proven Ops Strategies

This article walks through a real call‑center outage, outlines why traditional ad‑hoc debugging fails, and presents a structured approach—including symptom identification, rapid root‑cause isolation, enhanced monitoring, concise emergency playbooks, and intelligent automation—to dramatically reduce recovery time and move toward self‑healing operations.

automationcall centeremergency plan
0 likes · 13 min read
How to Speed Up Call Center Incident Resolution with Proven Ops Strategies
ITPUB
ITPUB
Oct 9, 2020 · Operations

How to Streamline Call Center Incident Management: Practical Steps and Best Practices

This guide walks through a real‑world call‑center slowdown incident, outlines common fault‑handling techniques, proposes monitoring enhancements, details a comprehensive emergency‑response plan, and introduces intelligent event‑processing concepts to help operations teams resolve outages faster and more reliably.

automationcall centeremergency response
0 likes · 15 min read
How to Streamline Call Center Incident Management: Practical Steps and Best Practices
Open Source Linux
Open Source Linux
Sep 12, 2020 · Operations

Mastering Incident Response: Core Principles and Practical Methods

This guide outlines essential incident‑response principles—prioritizing business restoration and timely escalation—while detailing practical methods such as restart, isolation, and degradation, and explains how to organize response teams and conduct thorough post‑incident reviews.

degradationfault handlingincident management
0 likes · 11 min read
Mastering Incident Response: Core Principles and Practical Methods
Efficient Ops
Efficient Ops
Sep 9, 2020 · Operations

Mastering Incident Management: Core Principles and Practical Methods

This guide outlines essential incident management principles—prioritizing business restoration and timely escalation—followed by detailed methodologies such as restart, isolation, and degradation, and explains role responsibilities, user impact handling, and post‑incident summarization for continuous improvement.

fault handlingincident managementoperations
0 likes · 10 min read
Mastering Incident Management: Core Principles and Practical Methods
Efficient Ops
Efficient Ops
Sep 8, 2020 · Operations

From Firefighting to Arson: Mastering Ops Availability in Three Stages

The article outlines a three‑stage ops maturity model—firefighting, fire prevention, and arson—explains how proactive fault‑injection drills, continuous availability improvements, and aligning technical metrics with business value can transform operations from reactive responders into strategic value creators.

availabilityfault injectionincident management
0 likes · 8 min read
From Firefighting to Arson: Mastering Ops Availability in Three Stages
Didi Tech
Didi Tech
Jun 3, 2020 · Backend Development

Stability Guidelines and Anti‑Patterns for Backend Services

Drawing on five years of incident reviews, the article defines a comprehensive stability framework for backend services—mandating timeout hierarchies, weak dependencies, service-discovery integration, staged gray releases, robust monitoring, capacity planning, and strict change management—while cataloguing common anti-patterns such as over-aggressive circuit breaking, static retries, improper timeouts, tight coupling, and insufficient isolation, and urging regular rehearsal of these practices.

backend stabilitydeployment best practicesincident management
0 likes · 21 min read
Stability Guidelines and Anti‑Patterns for Backend Services
Efficient Ops
Efficient Ops
Apr 12, 2020 · Operations

Master Incident Management: Definitions, Processes, and Best Practices

This guide explains fault management fundamentals—from ITIL‑based definitions and why it matters, to fault level classification, monitoring, emergency response, recovery, post‑mortem analysis, continuous improvement, and practical advice for practitioners—providing a comprehensive, actionable framework for reliable operations.

ITILcontinuous improvementfault handling
0 likes · 11 min read
Master Incident Management: Definitions, Processes, and Best Practices
Continuous Delivery 2.0
Continuous Delivery 2.0
Mar 6, 2020 · Operations

Google Incident Postmortem Checklist

The article presents a detailed Google‑derived post‑mortem checklist covering event data collection, root‑cause analysis, lessons learned, actionable improvement items, and review procedures to ensure systematic, non‑blame‑focused incident handling.

Checklistaction itemsincident management
0 likes · 5 min read
Google Incident Postmortem Checklist
360 Tech Engineering
360 Tech Engineering
Oct 31, 2019 · Operations

AIOps Implementation Practice at 360: Architecture, Models, and Automation

The article details 360's AIOps deployment, covering external speaker insights, internal architecture, data collection pipelines, AI models for resource recycling, alarm reduction, and correlation, as well as visualization dashboards, labeling platforms, and self‑healing mechanisms, illustrating a comprehensive AI‑driven operations framework.

AI monitoringAIOpsSelf-Healing
0 likes · 14 min read
AIOps Implementation Practice at 360: Architecture, Models, and Automation
dbaplus Community
dbaplus Community
Oct 16, 2019 · Operations

How to Cut Alert Noise: Practical SRE Strategies for Ops Teams

This article shares concrete SRE‑inspired techniques—duty‑roster scheduling, tiered alert handling, automation safeguards, dashboard focus on top‑3 alerts, time‑based filtering, and systematic code review—to dramatically reduce daily alarm volume while keeping on‑call teams motivated and effective.

SREalert optimizationincident management
0 likes · 15 min read
How to Cut Alert Noise: Practical SRE Strategies for Ops Teams
58 Tech
58 Tech
Jul 23, 2019 · Operations

Design and Implementation of an Open Alarm Platform for Monitoring Systems

The Open Alarm Platform provides a flexible data model, modular architecture, and robust stability features to enable various business lines to integrate their custom monitoring systems via APIs, offering alert convergence, merging, multi‑channel delivery, and comprehensive management while reducing development and maintenance costs.

System Designalertingincident management
0 likes · 9 min read
Design and Implementation of an Open Alarm Platform for Monitoring Systems
dbaplus Community
dbaplus Community
Jul 7, 2019 · Operations

Turning Online Incidents into Growth: From Firefighting to Real Technical Mastery

The article reflects on handling online incidents by first extinguishing the immediate problem, then digging into root causes, and expands the discussion to what truly constitutes technical ability, the pitfalls of reinventing solutions, raising one’s perspective, and the critical role of systematic retrospection.

Software Engineeringincident managementproblem solving
0 likes · 12 min read
Turning Online Incidents into Growth: From Firefighting to Real Technical Mastery
DevOps
DevOps
Jun 25, 2019 · Operations

Applying Emergency Room Principles to IT Operations: Kanban, Scrum, and Prioritization

The article draws parallels between emergency rooms and IT operations, describing how Kanban's WIP limits, one‑to‑one liaison models, transparent dashboards, and Scrum time‑boxing (daily stand‑ups and weekly reviews) help a globally distributed team prioritize urgent incidents while still advancing important non‑urgent work.

DevOpsIT OperationsPrioritization
0 likes · 10 min read
Applying Emergency Room Principles to IT Operations: Kanban, Scrum, and Prioritization
Efficient Ops
Efficient Ops
Oct 29, 2018 · Operations

How Youzan Manages Online Incidents: A Step‑by‑Step Guide

This article outlines Youzan's end‑to‑end online incident management process—from fault detection and coordination through root‑cause analysis, recovery, review, and actionable JIRA tracking—highlighting practical workflows, data analysis, and continuous improvement practices for reliable service delivery.

JIRA workflowfault handlingincident management
0 likes · 10 min read
How Youzan Manages Online Incidents: A Step‑by‑Step Guide
Efficient Ops
Efficient Ops
Sep 24, 2018 · Operations

How Checklist Thinking Fuels Ops Professionals' Lifelong Growth

This talk explores how ops engineers can achieve continuous professional development by adopting checklist thinking, covering growth drivers, error classification, practical checklist applications, cognitive models, and design principles that turn complex incidents into systematic, repeatable processes.

ChecklistDevOpsgrowth
0 likes · 34 min read
How Checklist Thinking Fuels Ops Professionals' Lifelong Growth
dbaplus Community
dbaplus Community
May 8, 2018 · Operations

How to Build Reliable Operations: From BCM to Google SRE Practices

This article examines the growing challenges of system availability in modern operations, explains the concept of availability and the N‑nine metric, introduces Business Continuity Management and Google SRE approaches, and provides concrete technical and managerial methods—including architecture standardization, scaling strategies, tooling, emergency drills, and incident‑centralized management—to improve operational reliability.

BCMSREavailability
0 likes · 30 min read
How to Build Reliable Operations: From BCM to Google SRE Practices
MaGe Linux Operations
MaGe Linux Operations
May 8, 2018 · Operations

What My First Day in Ops Taught Me About Mistakes and Teamwork

A personal account of switching jobs, the challenges of a first production shift, a critical menu‑click error, the team’s rapid response, and the lasting operational lessons learned about risk awareness, double‑checking, and continuous improvement.

Career TransitionLessons Learnedincident management
0 likes · 7 min read
What My First Day in Ops Taught Me About Mistakes and Teamwork
Efficient Ops
Efficient Ops
Jan 11, 2018 · Operations

Mastering Incident Troubleshooting: Proven SRE Strategies for Operations

This article shares practical SRE‑based principles for diagnosing and resolving online incidents, emphasizing systematic investigation, gathering clues, and prioritizing service restoration over immediate root‑cause identification to make troubleshooting less mystical and more effective.

SRETroubleshootingincident management
0 likes · 7 min read
Mastering Incident Troubleshooting: Proven SRE Strategies for Operations
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Dec 27, 2017 · Operations

Efficient Ticket System Operations During Double 11 Promotion

The article describes how a ticketing system with strict SLA enforcement, automated routing, and team‑based service management enabled rapid, orderly issue handling during the high‑volume Double 11 shopping event, achieving near‑90% resolution within 30 minutes and improving overall business stability.

Double 11SLAincident management
0 likes · 7 min read
Efficient Ticket System Operations During Double 11 Promotion
Architecture Digest
Architecture Digest
Nov 1, 2017 · Operations

A Structured Approach to Online System Issue Diagnosis and Recovery

This article outlines a systematic methodology for understanding, evaluating, and quickly resolving production system incidents by categorizing system layers, assessing impact, employing Linux diagnostic tools, and designing fault‑tolerant mechanisms to minimize downtime and maintain core functionality.

BackendLinux toolsPerformance Monitoring
0 likes · 12 min read
A Structured Approach to Online System Issue Diagnosis and Recovery
dbaplus Community
dbaplus Community
Oct 16, 2017 · Operations

How Ele.me Scaled Operations: Key Lessons from Incident Management and Capacity Planning

This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.

Capacity Planningincident managementmonitoring
0 likes · 14 min read
How Ele.me Scaled Operations: Key Lessons from Incident Management and Capacity Planning
Efficient Ops
Efficient Ops
Oct 9, 2017 · Operations

How Tencent Scales Operations for Holiday Traffic Surges

This article explains how Tencent's social platform operations team prepares for massive holiday traffic spikes by following a four‑stage process—business preparation, capacity evaluation, resource provisioning, and scaling with stress testing—while detailing team structures, operational standards, and the supporting tool ecosystem that enable reliable, high‑availability services.

Capacity Planningincident managementoperations
0 likes · 13 min read
How Tencent Scales Operations for Holiday Traffic Surges
ITPUB
ITPUB
May 15, 2017 · Operations

Mastering Online Incident Management: From Detection to Prevention

This article outlines a comprehensive methodology for handling large‑scale online service incidents, covering goals, the "jump‑fill‑avoid" framework, step‑by‑step processes for detection, diagnosis, remediation, and post‑mortem analysis, as well as essential monitoring, logging, and escalation infrastructure.

SREincident managementlog tracing
0 likes · 18 min read
Mastering Online Incident Management: From Detection to Prevention
ITPUB
ITPUB
Feb 21, 2017 · Operations

How We Resolved a Sudden DNS Outage That Took Down Our Website and App

When a Saturday early-morning outage left the company’s website and mobile app inaccessible for many users, the team traced the issue to an unpaid domain causing DNS resolution failures, detailed the investigation steps, temporary fixes, and lessons learned about DNS processes and operational readiness.

DNSincident managementoutage
0 likes · 13 min read
How We Resolved a Sudden DNS Outage That Took Down Our Website and App
Efficient Ops
Efficient Ops
Dec 28, 2016 · Operations

Transforming Financial Application Operations: Lessons from a European Rollout

This article shares a detailed case study of how a financial services team restructured European application operations, applied lean retrospectives, built a top‑down monitoring system, and introduced systematic stakeholder collaboration to dramatically improve incident response, system robustness, and user satisfaction.

DevOpsapplication monitoringincident management
0 likes · 14 min read
Transforming Financial Application Operations: Lessons from a European Rollout
Efficient Ops
Efficient Ops
Dec 19, 2016 · Operations

What 16 Major 2016 Outages Teach Us About Disaster Recovery

This article reviews sixteen notable 2016 service outages across finance, cloud, and entertainment, analyzes their causes—ranging from power failures to DDoS attacks—and highlights the critical need for robust disaster‑recovery and information‑security practices.

incident managementinformation securityoperations
0 likes · 11 min read
What 16 Major 2016 Outages Teach Us About Disaster Recovery
Efficient Ops
Efficient Ops
Nov 27, 2016 · Operations

When Ops Heroes Burn Out: Tackling Personal Heroism in Operations

The article explores personal heroism in operations, defining it as reliance on individual effort to keep flawed systems appearing normal, examines its short‑term benefits and long‑term drawbacks for companies, teams, and the heroes themselves, and offers practical strategies to eliminate this risky mindset.

SLAincident managementoperations
0 likes · 10 min read
When Ops Heroes Burn Out: Tackling Personal Heroism in Operations
Efficient Ops
Efficient Ops
Oct 23, 2016 · Operations

How Google’s SRE Postmortems Drive System Reliability

This article explains Google’s SRE postmortem philosophy, the criteria for writing postmortems, best practices for a blame‑free culture, and how collaborative knowledge‑sharing and incentives improve incident handling and overall system reliability.

SREincident managementoperations
0 likes · 14 min read
How Google’s SRE Postmortems Drive System Reliability
Efficient Ops
Efficient Ops
Oct 6, 2016 · Operations

How Ctrip Scales Application Operations: Practices, Automation, and Reliability

This talk details Ctrip's application operations framework, covering data‑center scale, multi‑application deployment on Windows, high availability goals, capacity‑prediction models, disaster‑recovery design, incident response, and the evolution from manual tooling to automated, intelligent operations.

Capacity PlanningCloud Computingautomation
0 likes · 21 min read
How Ctrip Scales Application Operations: Practices, Automation, and Reliability
Efficient Ops
Efficient Ops
May 17, 2016 · Operations

When a Single Cable Crashes a Network: Real Ops Incident Lessons

This article recounts two real‑world operations incidents—a network outage caused by an improperly configured portfast on a trunk link and an NFS failure that crippled an API service—then distills practical lessons on pre‑incident procedures, monitoring, fault handling, recovery, and post‑mortem practices.

Fault AnalysisITILNFS
0 likes · 11 min read
When a Single Cable Crashes a Network: Real Ops Incident Lessons
Efficient Ops
Efficient Ops
Feb 3, 2016 · Operations

Why Human Errors Still Plague Modern Ops and How to Prevent Them

This article examines recent high‑profile internet outages caused by human error, explores why operations teams are especially prone to mistakes despite automation and standards, and offers practical strategies—such as hiring the right people, fostering safety awareness, and turning professionalism into habit—to reduce future incidents.

Best Practicesautomationhuman error
0 likes · 14 min read
Why Human Errors Still Plague Modern Ops and How to Prevent Them
Efficient Ops
Efficient Ops
May 29, 2015 · Operations

Why Ctrip’s Outage Took Hours to Recover – Lessons for Ops Teams

The article examines Ctrip’s prolonged service restoration after a May 28 incident, analyzing the complexities of SOA‑based architectures, the pitfalls of black‑box operations, and how transitioning to white‑box, DevOps‑aligned practices can prevent similar outages.

DevOpsIT OperationsSOA
0 likes · 11 min read
Why Ctrip’s Outage Took Hours to Recover – Lessons for Ops Teams