Mastering SRE: Mindset, Monitoring, and Incident Response Strategies

Preface

Stability is no longer limited to peak‑time guarantees; it has become a systematic practice. Drawing on two years of SRE work at Hema and broader experience across Alibaba, the author shares perspectives on SRE mindset, monitoring, incident response, resource management, and promotion mechanisms.

1. Mindset & Attitude

Who should do stability?

Responsible people : proactively respond to alerts, tickets, and risks.

Avoid newcomers : inexperienced staff lack business knowledge and may miss risks.

Don’t pick overly "obedient" people : they may not proactively improve stability.

How teams should support SRE

Provide resources : stability is a team effort, not just the SRE’s job.

Give space : allow SREs to think, innovate, and link stability work with architecture upgrades.

Clarify responsibility : distinguish whether an issue stems from stability work, team negligence, or business changes.

2. Monitoring

Effective monitoring is the "eyes" of an SRE, enabling rapid detection of anomalies across service, data, and financial dimensions.

Five dimensions of monitoring

Service health (QPS, latency, error rate)

Data correctness

Financial loss prevention

System‑level metrics (load, resource usage)

Dependency health (downstream services, databases)

Monitoring dashboard

A concise dashboard should display core business entry QPS/RT, top error codes, order volume trends, key downstream dependencies, and any custom stability indicators.

Alert management

Use phone alerts sparingly for critical, large‑scale incidents.

Maintain a single DingTalk alert group per team to centralize notifications.

Persist all alerts for post‑mortem analysis.

Limit daily alert volume (e.g., < 100 alerts in normal operation) by adjusting thresholds and using composite rules.

3. Incident Response

When a fault occurs, an SRE should stay calm, initiate a conference call, and follow a structured response flow:

Confirm who is handling the alert.

Quickly locate the problem scope.

Estimate impact range and provide data.

Present decisions needed from leadership and suggest actions.

Report progress and recovery status.

After resolution, conduct a thorough post‑mortem covering timeline, root cause, mitigation steps, and improvement actions.

4. Resource Management

SREs must ensure sufficient capacity while avoiding waste. Core services should keep load around 1‑1.5 with no more than ten instances; non‑core services may tolerate 1.5‑2 load. Maintain a 20% buffer for machines, caches, databases, and messaging systems to handle growth or emergencies.

Maintain a resource inventory that records current usage, limits, and pressure points for machines, storage, and databases.

5. Collaboration with Peer Teams

Never mock other teams when they encounter incidents.

Avoid complacency; always verify your own stability.

Do not claim invulnerability; stay vigilant.

Conclusion

By cultivating the right mindset, establishing systematic monitoring, managing alerts wisely, and maintaining disciplined resource control, SREs can transform stability from a reactive “clean‑up” role into a proactive, value‑adding discipline.