Mastering Incident Management: Principles and Methods for Effective Fault Handling

1. Fault Handling Principles

The principles for handling incidents are twofold: prioritize business recovery and timely escalation.

1.1 Business Recovery First

Regardless of the situation or fault level, the primary goal is to restore business operations, not merely locate the root cause. For example, when Application A fails to call Application B, two approaches can be taken:

Method 1: Investigate the failure path between A and B, identify the problematic component (e.g., HA connection issue), and restart or scale it.

Method 2: From A's server, ping B's network; if the port and network are reachable, directly bind B's host entry.

Typically, Method 2 is faster, but if A and B span data centers, Method 1 may take longer yet still restores service promptly.

1.2 Timely Escalation

When an incident occurs, its impact can only be roughly predicted, so it must be escalated to leadership promptly to secure resources. Escalation is required in cases such as:

Clear business impact (e.g., PV, UV, cart, order, payment metrics).

Critical business alerts (e.g., core services, essential components).

Significantly prolonged resolution time.

Attention from senior leadership, monitoring centers, or customer service.

Issues beyond the responder's capability.

Note: The operations leader must be the first to know about any incident; otherwise, the responder is considered negligent.

2. Fault Handling Methodology

Incident handling is generally divided into three stages: pre‑incident, during incident, and post‑incident . This article focuses on methods used during the incident phase.

2.1 Service‑Centric Handling Methods

From a service perspective, the three most important recovery actions are restart, isolation, and downgrade .

Restart: Includes service restart and OS restart. The typical order is the faulty object, then its upstream, then downstream components.

Example: For a RabbitMQ failure, restart RabbitMQ first; if ineffective, restart the upstream producer, then the downstream consumer.

Important: Do not skip restart just because metrics look normal; the goal is business restoration, not root‑cause analysis.

Isolation: Remove the faulty component from the cluster so it no longer provides service. Common methods:

Set upstream weight to zero or stop the service if health checks exist.

Bind hosts or adjust routing to bypass the faulty component (e.g., disable a specific route).

Purpose: Prevent cascade failures.

Downgrade: Implement a fallback plan to avoid larger failures. Downgrade is not the optimal user experience but maintains service continuity (e.g., alternative payment channels).

Downgrade requires coordination with development teams and should be part of a pre‑planned strategy.

2.2 Impact‑Centric Handling Methods

Incidents affect two user groups: external users and internal users.

External Users

Steps:

Reproduce the issue locally; if reproducible, treat it as an internal problem.

If not reproducible, involve other internal users to rule out environment issues, and ask the external user to bind hosts or bypass DNS to verify external network problems.

If both fail, collect external user information (e.g., outbound IP, client version) using a template to reduce communication overhead.

Internal Users

Includes internal application calls and internal staff reports; handle these using the methods described in 2.1.

2.3 Organizational Structure During Incident

Effective incident response typically involves three roles acting simultaneously:

Incident Responder: Focuses on rapid business restoration.

Incident Investigator: Steps in when the responder’s methods fail, to find the root cause.

Communicator: Relays accurate information within the team and to external stakeholders.

In practice, not all roles are present at once (e.g., during night shifts only a responder may be on duty). Roles can be reused as needed.

3. Post‑Incident Summary

Each incident requires a thorough post‑mortem to address root causes, prevent recurrence, and apply PDCA (Plan‑Do‑Check‑Act) for continuous improvement.

When documenting the summary, consider responsibility attribution and handle difficult stakeholders carefully.

Downgrade should be viewed as the last resort to keep services alive.