Reducing MTTR: Monitoring, Fast Incident Response, and Team Practices

When a system failure occurs, MTTR (Mean Time To Repair) is a crucial metric that measures the average time needed to restore service; long MTTR can cause severe business impact such as transaction loss.

MTTR is calculated by dividing the total maintenance time by the number of maintenance actions within a given period, as illustrated in the formula image.

Key steps to shorten MTTR:

Problem discovery time: Build an alert system (e.g., UMP) with thresholds for availability, TP99, and call volume to quickly identify issues and reduce noise.

Mitigation time: Establish a fast‑stop‑bleed mechanism (DUCC switch, rollback, traffic limiting) and a well‑practiced emergency response process.

Intelligent analysis: Use tools such as distributed tracing (Pfinder), performance dashboards, and log analysis to pinpoint bottlenecks.

Team roles and workflow: Define a fault commander, communication lead, and executors; conduct regular drills, on‑call rotations, and clear escalation paths.

Alarm configuration: Apply a "tight‑then‑loose" strategy, prioritize critical alerts, and ensure alerts are accurate and actionable.

Post‑incident review: Document the incident, analyze root causes, and create a COE (Center of Excellence) report to prevent recurrence.

Example alert configuration (critical and warning levels):

critical告警方式：咚咚、邮件、即时消息(京ME)、语音
可用率：（分钟级）可用率 < 99.9% 连续 3 次超过阈值则报警，且在 3 分钟内报一次警。
性能：（分钟级）TP99 >= 200.0ms 连续 3 次超过阈值则报警，且在 3 分钟内只报一次警。
调用次数：当方法调用次数在 1 分钟的总和，连续 3 次大于 5000000 则报警，且在 3分钟内只报一次警
warning告警方式：咚咚、邮件、即时消息
可用率：（分钟级）可用率 < 99.95% 连续 3 次超过阈值则报警，且在 30 分钟内报一次警。
性能：（分钟级）TP99 >= 100.ms 连续 3 次超过阈值则报警，且在 30 分钟内只报一次警。
调用次数：当方法调用次数在 1 分钟的总和，连续 3 次大于 2000000 则报警，且在 3 分钟内只报一次警

Additional best practices include: Collect and compare input/output parameters to verify issues. Retain a live instance for post‑mortem analysis. Provide timely feedback even when progress is minimal. Use the "three‑character mantra" (三字经) to remember the order: report, coordinate, act.

By following these guidelines, teams can reduce MTTR, improve system stability, and deliver higher reliability to end users.