Mastering Incident Response: Core Principles and Practical Methods

1. Fault Handling Principles

The principles for handling incidents are twofold:

Prioritize restoring business operations.

Escalate promptly.

1.1 Prioritize Restoring Business

Regardless of the situation or severity, the first goal is to get the business back online, which differs from fault diagnosis. For example, if Application A fails to call Application B, two approaches exist:

Method 1: Investigate the failure path between A and B, identify the problematic component (e.g., HA connection issue), and restart or scale it.

Method 2: From A’s server, ping B’s network; if the port and network are reachable, directly bind B’s server in the hosts file.

Typically, Method 2 is faster, but if A and B span data centers, Method 1 may take longer; despite disrupting architectural balance, it restores service quickly, embodying the “business‑first” principle.

1.2 Timely Escalation

When an incident occurs, its impact can only be roughly predicted, so it must be escalated to leadership promptly to provide real‑time information and coordinate resources. Escalation is required when any of the following conditions are met:

Clear business impact such as fluctuations in PV, UV, cart, order, or payment metrics.

Critical alerts for high‑importance services or core components.

Processing time exceeds defined thresholds.

Senior leadership, monitoring centers, or customer support have already noticed the issue.

The problem is clearly beyond the responder’s capability.

Note: The operations leader must be the first to know about any incident; learning about it from another team indicates a failure in the response process.

2. Fault Handling Methodology

Incident handling is generally divided into three phases: pre‑incident (analysis), during‑incident (resolution), and post‑incident (review). This article focuses on methods used during the incident phase.

2.1 Service‑Centric Operational Methods

From a service perspective, the three most important recovery actions are restart, isolation, and degradation .

Restart : Includes service restart and OS restart. In an incident, any component can be restarted. The typical order is: affected object → upstream components → downstream components, with farther components restarted later.

Example: For a RabbitMQ failure, first restart RabbitMQ; if ineffective, restart the upstream producer; if still unresolved, restart the downstream consumer.

Important: Do not skip a restart simply because metrics look normal; the goal is to restore service, not to diagnose.

Isolation : Remove the faulty component from the cluster so it no longer provides service. Common techniques, ordered by frequency, are:

Set upstream weight to zero or stop the faulty service if health checks exist.

Bypass the component via hosts binding or routing changes (e.g., disabling a specific line in a smart routing system).

Purpose: Prevent cascade failures (avalanche effect).

Degradation : Implement a fallback plan to avoid larger failures. Degradation is never the optimal user experience; it may affect payment flows or other business processes, but it keeps the system functional.

Degradation requires coordination with development teams; a pre‑plan should include quick domain switching, retry mechanisms, and the ability to disable retries to protect upstream services.

Objects should be stateless whenever possible; if stateful, ensure idempotency for retries. Production objects typically fall into three categories:

Stateless (majority).

Temporarily stateful (needs remediation).

Stateful (minority).

2.2 Impact‑Based Operational Methods

After an incident, impact is classified as external or internal users.

2.2.1 External Users

External user handling aims to convert external issues into internal ones. Steps include:

Reproduce the problem locally; if reproducible, it is an internal issue.

If not reproducible locally, have other internal users test, and ask the external user to bind hosts or bypass DNS to rule out network problems. Successful access after hosts binding indicates an external issue.

If both attempts fail, gather detailed external user information (e.g., exit IP, client version) to proceed.

2.2.2 Internal Users

Internal issues (application‑to‑application calls or internal staff reports) are addressed using the methods described in section 2.1.

2.3 Organizational Structure During Incident Handling

Effective incident response typically involves three roles acting simultaneously:

Incident Responder – focuses on restoring service quickly.

Incident Investigator – steps in when the responder’s methods fail or when root‑cause analysis is needed.

Communicator – ensures accurate information flow within the team and to external stakeholders.

In practice, especially during off‑hours, only the responder may be active; the investigator conducts post‑incident analysis the next day. Roles can be combined as needed.

3. Incident Post‑Mortem

Post‑mortem analysis is crucial; each incident should be examined to identify root causes, prevent recurrence, and drive continuous improvement (PDCA cycle).

When documenting post‑mortems, pay special attention to responsibility attribution and handling difficult stakeholders.

Degradation, as a last‑resort safety measure, must be carefully managed.