A coordinated overseas cyber‑attack breached a Guangzhou tech firm's self‑service equipment backend, causing hours of service outage, data leakage, and significant losses, prompting swift police investigation, evidence preservation, and a detailed technical analysis of the attackers' methods.
Based on analysis of over 500 enterprise server failure cases, this guide lists 20 absolutely prohibited server actions across six dimensions, each illustrated with a real incident and practical technical measures to prevent recurrence.
This article shares a step‑by‑step account of taking over a content‑risk stability role in early 2024, defining system stability, diagnosing recurring issues, and implementing a three‑phase framework—pre‑emptive reduction, impact mitigation, and post‑incident improvement—to boost success rates, cut incident response time, and achieve a modular architecture.
This article explains the importance, pain points, architecture, core capabilities, and future roadmap of the 360 Zhihui Cloud "Yunzhou" unified alert service, showing how it improves observability, reduces alert noise, and accelerates incident response for modern cloud‑native systems.
This guide explains the significance of business monitoring, differentiates technical and business metrics, outlines a step‑by‑step process for building a robust business indicator system, and shares practical methods, tools, and common pitfalls to ensure reliable, actionable monitoring in operations.
A recent case where a maintenance worker exploited device‑management flaws to steal confidential files for foreign spies highlights the need for heightened vigilance, strict self‑discipline, and prompt reporting, offering practical steps to safeguard against similar security breaches.
This article presents a detailed walkthrough of constructing a robust security architecture, covering single‑person security team strategies, risk perception and quantification, rapid incident response, automated detection, precise strike mechanisms, deterrence tactics, and forward‑looking plans for intelligent, data‑driven risk management.
Many companies deploy fragmented monitoring and observability tools yet still struggle to pinpoint incidents; this article analyzes the root causes—under‑utilized tools and scenario‑agnostic data—and offers practical steps to organize metrics, build layered insights, and improve fault‑resolution efficiency.
This article details a real‑world Linux server breach, describing the symptoms, investigative commands, log analysis, malicious script removal, file attribute unlocking, and practical remediation steps, while highlighting key lessons and preventive measures for future security.
In December 2024, Xiaohongshu’s information security team topped the Shanghai “Panshi Action” competition, earning top blue‑team honors and a zero‑loss defense record by leveraging real‑time traffic monitoring, big‑data analytics, rapid incident response, and successful attacker attribution.
This article documents a step‑by‑step investigation of a compromised Linux server that exhibited 100% CPU usage, detailing process, network, and startup‑service analysis, the discovery of a cryptomining malware, and the complete removal procedure.
This article describes the design and implementation of a lightweight cloud‑native alert platform, outlining its core features, future enhancements, system architecture, and demo screenshots, offering practical insights for SREs and operations teams handling growing monitoring workloads.
A disgruntled former infrastructure engineer at a U.S. industrial firm deleted backups, locked administrators, and demanded $750,000 in Bitcoin, leading to his arrest and highlighting the severe risks, legal consequences, and mitigation strategies associated with insider ransomware threats.
This article explains a security vulnerability grading standard that defines five severity levels (S0‑S4), outlines handling timeframes, describes conditions for automatic level upgrades, and lists typical vulnerability types for each level to guide effective risk management.
This article details XiaoBai's journey from struggling with ad‑hoc incident handling to designing a comprehensive platform that captures anomaly data, diagnoses root causes, and enables every team member to respond quickly and consistently, ultimately achieving a "everyone can respond" operation model.
This article shares ten practical operations strategies—including change rollbacks, safe handling of destructive commands, prompt customization, rigorous backup and verification, production environment discipline, careful handovers, robust alerting, cautious automatic failover, meticulous checks, and simplicity—to dramatically improve system reliability and availability.
This article recounts how a solo developer built a Django‑based Docker image signing service to meet PCI DSS requirements, faced two severe incidents—including a 17.5‑hour P0 outage caused by concurrency limits and a misconfigured Rekor service—and shares the operational lessons learned for reliable SRE practice.
The massive Microsoft blue‑screen outage caused by a faulty CrowdStrike update highlights the dangers of single‑system reliance, poor code quality, insufficient QA, and the need for staged rollouts, robust backup, real‑time monitoring, and proactive incident‑response strategies for modern IT organizations.
This article outlines comprehensive system stability strategies for backend development, covering technical design reviews, key reliability techniques such as rate limiting, circuit breaking, timeout handling, isolation, and deployment safeguards like monitoring, gray releases, and rollback, aiming to reduce incidents and improve operational resilience.
This article examines the critical CVE‑2024‑6387 OpenSSH Server 0‑day vulnerability, explains its exploitation mechanics, and outlines effective emergency response strategies, including JD Cloud’s security operations solutions, to help enterprises swiftly mitigate risks, manage attack surfaces, and strengthen overall information security posture.
This article explains why regular disaster role‑playing, systematic testing, and focused responder preparation are essential for building robust incident response capabilities and reducing operational risk in production environments.
This article demystifies Site Reliability Engineering (SRE) by explaining its origins, core responsibilities, essential skill sets, and key practices such as observability, incident response, testing, capacity planning, automation, user support, on‑call duties, and the definition of SLI/SLO/SLA, providing a comprehensive guide for modern operations teams.
A quiet Saturday turned into a disaster when a simple UPDATE query accidentally deleted every order in production, prompting a rapid, step‑by‑step recovery, a post‑mortem analysis of the root causes, and a set of hard‑won operational lessons for any engineering team.
A collection of firsthand accounts reveals how seemingly harmless actions—changing system time, mistyping a script name, accidental deletions, and reckless debugging—triggered large‑scale service disruptions, forced emergency rollbacks, and costly penalties, highlighting the high stakes of operational negligence.
On April 8, a Tencent Cloud API outage caused console login failures for nearly 2,000 customers, affecting several dependent services for 87 minutes, and the detailed root‑cause analysis and subsequent improvement actions are presented to enhance system resilience and change management.
On April 8, 2024, a cloud API failure disrupted Tencent Cloud's console for 87 minutes, affecting 1,957 customers, prompting a detailed incident review that uncovered version‑compatibility and configuration‑data issues and led to a set of operational improvements.
After a weekend support ticket led to a reckless UPDATE that erased all orders in a production PostgreSQL database, the author details the rapid recovery steps, analyzes the human errors behind the disaster, draws lessons from Chernobyl, and outlines concrete post‑mortem improvements to prevent future data loss.
This report details a network operations incident where a BGP routing change caused an EBGP neighbor to go idle, outlines the step‑by‑step troubleshooting, analysis of the root cause, and the implemented solution involving a new L3 node and redundant EBGP peers.
This guide walks ops teams through real‑world incident scenarios, from quick symptom identification and emergency recovery to improving monitoring, crafting concise emergency plans, and leveraging automation for smarter fault handling, helping organizations restore services faster and reduce downtime.
This article explores how HuoLala's host security HIDS leverages Neo4j graph databases and the Neovis.js visualization library to unify process, network, and file data, enabling rapid attack‑chain reconstruction, efficient multi‑cloud incident response, and improved security operations.
A developer mistakenly set a cloud disk snapshot to public, exposing a major client’s data, and recounts the frantic rollback, the ensuing panic among teammates, and the hard‑won operational lessons about high‑risk manual tasks, proper safeguards, and the need for visualized tooling.
Drawing on twenty years of Google SRE experience, this article outlines eleven practical lessons—from scaling mitigation to disaster‑resilience testing—that help teams design, operate, and evolve reliable large‑scale services.
D‑Eyes is an open‑source detection and response tool from NSFOCUS that runs on Windows and Linux, offering command‑line utilities to scan files, processes, host information, network connections, and perform baseline and software‑supply‑chain checks, with built‑in YARA rules for ransomware, mining malware, botnets, and webshells.
The author recounts a 2018 incident where a cloud disk snapshot was unintentionally made public, exposing customer data, and shares a detailed reflection on the operational mistakes, risk management failures, and recommended safeguards for high‑risk cloud operations.
This article outlines a comprehensive data fault governance framework that classifies metrics, defines three development phases, establishes fault‑grading standards, clarifies responsibilities across development, data‑warehouse, and analytics teams, and implements pre‑, during‑, and post‑incident safeguards to dramatically reduce fault frequency and recovery time.
A cloud engineer mistakenly set a disk snapshot to public, exposing a major client’s data, rushed a rollback, and then reflected on the root causes, highlighting the need for strict review, visual tools, and risk‑aware practices in high‑risk operations.
This article outlines the red‑blue adversarial testing process for a big‑data platform during the Double‑Eleven promotion, detailing its purpose, benefits, step‑by‑step execution, common issues, and recommendations to improve system reliability and security.
The article explains how a configurable yellow banner system can be used on web interfaces to promptly inform users about service disruptions, guide their actions, increase transparency, improve experience, and outline implementation considerations such as configurability, persistence, and independent deployment.
This article explains the concept and importance of MTTR (Mean Time To Repair), shows how to calculate it, and provides a comprehensive set of monitoring, alerting, rapid mitigation, tool‑assisted analysis, and team coordination techniques to significantly shorten incident resolution time and improve system reliability.
This article details Bilibili's data quality assurance framework, covering its evolution across four data platform stages, the architecture of its quality data warehouse, core capabilities such as a complete assurance system, digital‑driven continuous optimization, and efficient incident handling, plus case studies, future plans, and a Q&A session.
This article walks through Yuque’s October 23 service disruption, detailing each timeline milestone, analyzing the root causes, highlighting the importance of monitoring and data integrity checks, and offering concrete post‑mortem recommendations to improve future incident handling.
The article details Bilibili’s comprehensive alert‑management platform—its background, cloud‑vs‑self‑built solution comparison, closed‑loop design, distributed architecture, rule configuration, noise‑reduction, automated root‑cause analysis, and governance practices that cut weekly alerts from 1,000 to under 80, while outlining future enhancements.
Didi’s multi‑year data‑stability program for its customer‑service platform progressed through fault‑centered engineering, business‑aligned cross‑team work, and capability normalization, instituting pre‑, mid‑ and post‑fault safeguards, clear ownership, automated alerts and repair tools, which cut fault count by 42 % and more than doubled mean‑time‑to‑repair while boosting team communication and satisfaction.
This article details a real‑world Linux server intrusion, describing the observed symptoms, the forensic investigation using commands like ps, top, last, and grep, the removal of malicious cron jobs and backdoors, and the lessons learned for securing SSH, file attributes, and cloud security groups.
A self‑built Kubernetes cluster was compromised when an unprotected node with empty iptables and a kubelet that allowed anonymous API access was hijacked for Monero mining, prompting a detailed post‑mortem, root‑cause analysis, and hardening recommendations.
The article outlines fifteen essential cybersecurity metrics—thirteen process indicators such as mean detection and response times, and two result indicators like data loss incidents and security ROI—to help organizations evaluate, monitor, and improve their security posture and inform investment decisions.
The recent Xiaohongshu app crash sparked public outcry and a viral screenshot blaming a developer, but the article explains that software bugs are inevitable, responsibility lies with the whole team, and proper debugging, testing, and process improvements are the rational response.
The article outlines Tencent Search’s stability governance, detailing a multi‑layered availability architecture, disaster‑recovery mechanisms, precise monitoring, rapid emergency workflows, pre‑release interception, extensive automation, and a collaborative governance model that together enhance system resilience, incident detection, and swift remediation.
This article details a November 2022 database outage caused by a sudden CPU and load surge, explains how the team diagnosed the issue, outlines the emergency steps taken, and shares practical SQL performance optimization recommendations to prevent similar incidents.
This article details Huolala's journey in establishing a comprehensive technical stability framework, covering organizational challenges, risk governance, incident response, cultural initiatives, and future automation to enhance system reliability at scale.
Hello’s technical‑risk team created a comprehensive emergency‑response system for large‑scale promotions—prioritizing core scenarios, running high‑frequency drills, modeling fault‑portraits, defining metric‑based triggers and clear rollback actions—which delivered zero incidents during the 930 Big Sale, achieved over 80 % core‑line coverage, and now aims to automate plan selection and execution.
Robusta is a Python‑based open‑source platform that layers on top of monitoring stacks like Prometheus to automatically detect, diagnose, and remediate Kubernetes alerts through built‑in automations, optional web UI, and Helm‑based installation for cloud‑native environments.
This article details a recent server intrusion case, describing the observed symptoms, possible causes, step‑by‑step forensic investigation using commands like ps, top, grep and crontab, and comprehensive remediation actions such as tightening SSH security, unlocking and restoring system binaries, removing malicious scripts, and key lessons for future protection.
This article explains what stability assurance is, outlines a systematic workflow—including anomaly identification, monitoring configuration, impact assessment, and solution planning—and provides practical methods such as capacity estimation, traffic limiting, load testing, scaling, and pre‑heating to ensure services remain stable during both daily operations and high‑traffic events.
This guide explains essential incident response roles, how to quickly identify the source of a problem, prioritize actions, use efficient communication tools, and address human factors to improve your team's emergency response capabilities.
This article provides a step‑by‑step technical analysis of the DarkKomet remote‑access trojan, covering its capabilities, infection vectors, detection methods using TTP‑driven EDR, containment actions, eradication procedures, root‑cause forensics, and post‑incident recovery measures.
A senior engineer accidentally issued a DELETE request on an ElasticSearch index holding 17 million product records, triggering a massive data loss incident, and the team’s subsequent recovery strategies, scaling challenges, and process improvements are detailed to guide backend developers.
This article explains why nightly disk‑space alerts demand automated fault‑self‑healing, outlines the necessary process standards, describes monitoring platform dimensions, details a multi‑source self‑healing platform with CMDB integration, and provides practical options for script execution and result notification.
This article explains what a reliability culture is, why it matters, how to cultivate it through mission statements, early‑stage reliability testing, chaos‑engineering practices like GameDays and FireDrills, and how organizations can continuously learn from incidents to improve system availability and customer trust.
This article details a real‑world Linux server compromise, describing the symptoms, possible causes, investigative commands, hidden malicious scripts, file attribute locks, and practical remediation steps to restore the system and improve future security.
From kickoff meetings and traffic forecasting to load‑testing strategies, rate‑limiting designs, emergency runbooks, and post‑event retrospectives, this guide walks engineers through the complete technical workflow required to ensure a Double‑11‑scale e‑commerce promotion runs smoothly and safely.
This guide walks you through a complete Linux incident‑response workflow—identifying suspicious behavior, locating and terminating malicious processes, eliminating virus files, closing persistence mechanisms, and hardening the system to prevent future compromises—using practical shell commands and real‑world examples.
This post‑mortem details the July 2021 Bilibili outage caused by a Lua bug in the OpenResty‑based SLB, describing the timeline, root‑cause analysis, mitigation steps, and the technical and organizational improvements implemented to prevent similar incidents.
This article details the July 13, 2021 Bilibili service outage caused by a Lua‑based SLB CPU spike, describing the incident timeline, root‑cause analysis of a weight‑zero bug, mitigation steps including new SLB deployment, and the subsequent operational and architectural improvements.
On July 13 2021 Bilibili’s L7 SLB crashed when a recent Lua deployment set a balancer weight to the string “0”, producing a NaN value that triggered an infinite loop and 100 % CPU, prompting emergency restarts, a fresh cluster rollout, and long‑term safeguards such as automated provisioning, stricter Lua validation, and enhanced multi‑active disaster‑recovery processes.
This article details the challenges, design choices, deployment steps, detection model creation, data processing, visualization, and future plans of JD Daojia's security operations platform, highlighting the use of Graylog, Elasticsearch, and MongoDB to achieve scalable, real‑time threat detection and response.
The article examines why SRE teams experience overload due to high incident response demands, analyzes contributing factors such as production issues, alert volume, and manual processes, and proposes comprehensive mitigation steps including better testing, load management, and proactive error detection to reduce on‑call burden.
This article chronicles the discovery of a server breach used for cryptocurrency mining, detailing the malicious code, forensic analysis of the trojan's actions and artifacts, and the step‑by‑step remediation measures taken to secure the system.
This article chronicles the discovery of a server breach used for cryptocurrency mining, analyzes the malicious Python payload and its system modifications, and provides concrete remediation steps such as system reinstall, non‑root deployment, firewall hardening, and Nginx authentication.
This article explains how limited resources shape product requirement prioritization, test‑bug grading, product‑module classification, online bug severity, and incident response levels, offering practical frameworks and concrete grading tables to help teams make objective, value‑driven decisions throughout a product’s lifecycle.
To tackle growing production complexity and past incident delays, the consumer trading platform introduced a three‑tier NOC‑SLA with intelligent baselines powered by Facebook Prophet, streamlined alert rules, and an SOS‑linked workflow, boosting detection frequency, cutting critical response times to under five minutes, and improving overall system reliability while emphasizing ongoing baseline and rule maintenance.
Bilibili’s SRE team combines stability theory, detailed fault‑stage and operational metrics, and a unified emergency‑response platform—including on‑call scheduling, fault‑command incident commanders, automated fault portraits, and rapid post‑mortems—to transform frequent incidents into data‑driven, collaborative recoveries and lay groundwork for AI‑assisted self‑healing.
Wireless operations and maintenance (O&M) evolved from backend‑focused practices to address stability and performance of mobile‑device services, tackling low issue detection rates and delayed responses through improved monitoring, gray‑release tagging, phased rollouts, AI‑driven diagnostics, and automated release gates, while inviting collaborative development.
The article reviews ten major 2021 service outages—from Chinese platforms like Bilibili and Futu to global giants such as Facebook, Roblox, and AWS—analyzing their root causes, redundancy failures, and the operational lessons needed to prevent future black‑swans.
A Kubernetes node was compromised for Monero mining due to empty iptables, an exposed kubelet API, and a mis‑commented configuration, prompting a detailed post‑mortem and practical hardening steps to prevent similar attacks.
The article presents a comprehensive case study of KeMonitor, a one‑stop monitoring and alert platform built by 贝壳找房 to unify fragmented alerts, define lifecycle‑based governance, standardize alert metadata, implement graded subscription, on‑call escalation, silencing, self‑healing, and post‑mortem analysis, thereby improving incident response efficiency and reducing alert fatigue.
A Kubernetes cluster was compromised when a misconfigured kubelet allowed anonymous API access, enabling attackers to run a Monero miner; the post details the investigation, root causes, and practical hardening steps to prevent similar intrusions.
A comprehensive review of ten major 2021 internet outages—from domestic platforms like Bilibili and Futu to global services such as Facebook, Roblox, and AWS—examines their root causes, the role of infrastructure design, and the operational lessons needed to improve system resilience.
Huawei Cloud’s internal “blue‑team” launched over twenty coordinated attacks around Chinese New Year, but the company’s SRE “red‑team” and a dedicated 24/7 “special forces” unit detected, isolated, and resolved incidents within minutes, keeping failure rates below 0.01% and demonstrating advanced cloud operations and security practices.
A self‑built Kubernetes cluster suffered a crypto‑mining intrusion due to empty iptables and a misconfigured kubelet, prompting a detailed post‑mortem that outlines the symptoms, root‑cause analysis, and practical hardening steps to protect similar environments.
This guide walks through a four‑stage Linux incident‑response workflow—identifying symptoms, killing malicious processes, closing persistence mechanisms, and hardening the system—while providing the exact shell commands needed to detect and eradicate Linux malware.
This guide outlines the typical Linux and Windows environments encountered in security incidents, common threats such as mining and ransomware, and provides a step‑by‑step workflow with essential commands for process, user, network, and file investigation to identify and remediate compromises.
After discovering a suspicious process on one of our self‑built Kubernetes nodes, we traced the intrusion to a misconfigured kubelet that exposed the API, allowing attackers to run a Monero mining script, and we outline the investigation steps and hardening measures to prevent similar breaches.
In Xi'an's Lianhu District, a disgruntled former network administrator exploited self‑taught networking skills to illegally infiltrate a hospital's internal servers, remotely executing destructive operations that deleted critical files, disabled printers, CT and ultrasound machines, and ultimately caused the entire medical information system to collapse, prompting a police investigation that led to his arrest and criminal detention for damaging computer information systems.
This article examines real-world database deletion incidents, the associated legal consequences, and how DevOps culture and operational best‑practices can turn such mistakes into learning opportunities rather than career‑ending failures.
Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.
This article details how a major securities firm analyzed business stability, built a comprehensive stability engineering platform using chaos engineering, practiced extensive fault‑injection drills, and outlines future directions such as random‑scenario exercises, red‑blue battles, and AI‑driven risk detection.
This guide explains how to design and practice a structured incident‑response process—defining problems, applying quick‑recovery steps, analyzing root causes, standardizing solutions, and assigning critical roles—to dramatically reduce production outage duration.
This guide outlines a structured approach to incident response, detailing problem definition, temporary fixes, root‑cause analysis, solution design, implementation, and standardization, while highlighting four critical roles—commander, communicator, rapid‑recovery lead, and diagnosis lead—to ensure swift, coordinated recovery of production services.
This guide walks through common Linux emergency scenarios—such as mining malware, ransomware, and backdoors—detailing a step‑by‑step workflow and providing essential command‑line tools for process, user, network, and file investigation on CentOS 6 and Windows Server 2008 systems.
When a server suddenly generated 800 MB of outbound traffic and SSH became unresponsive, the author traced the issue to a hidden backdoor, blocked the malicious IP, identified compromised binaries, removed malicious processes and startup scripts, and outlined preventive security measures.
This article presents a comprehensive, step‑by‑step framework for guaranteeing system reliability during high‑traffic promotional periods, covering SRE hierarchy, stability criteria, profiling, monitoring, capacity planning, incident response, and post‑event analysis to help teams build resilient services.
This article recounts a production server data loss caused by an erroneous rm -rf command, describes how ext3grep and extundelete were used to attempt file restoration, and explains how MySQL binlog replay finally recovered critical data while sharing lessons learned for future incident handling.
This article shares a step‑by‑step Linux incident‑response workflow, including an automated Bash information‑gathering script, analysis of malicious cron jobs and a 439‑line mining malware, its SSH‑based lateral spread, and practical cleanup procedures with a reusable toolbox on GitHub.
This article walks through a real‑world Linux mining malware infection, detailing how the attacker hid a malicious cron job, used LD_PRELOAD rootkits, propagated via SSH keys, and how the analyst uncovered and removed the threat using busybox, strace, and careful forensic commands.
iQIYI’s SOAR platform, built on StackStorm and the Walkoff visual editor, integrates security components, scripts, chat‑ops bots, and a mini‑program to automate detection and response, cutting MTTR by roughly 75% across high‑frequency routine tasks and low‑frequency critical incidents while planning broader coverage and knowledge‑base expansion.
This article examines the critical role of stability governance in evolving systems, outlines a three‑stage framework—usability, monitoring alerts, and online emergency—illustrated with a case study of an electronic waybill service, and shares concrete strategies for prevention, detection, response, and post‑mortem to achieve predictable, observable, and fast‑acting reliability.
This article analyzes the concept of stability governance, outlines its five fault‑management sub‑domains, examines the pain points of an electronic waybill service, and presents a comprehensive three‑phase strategy—prevention, perception, reach, mitigation, and post‑mortem—backed by concrete implementation steps in availability, monitoring, and online emergency handling.
This article explains how Netflix built the Telltale monitoring system to consolidate data sources, provide multidimensional health assessments, deliver intelligent alerts, and streamline incident management for over 100 production applications, reducing on‑call fatigue and improving service reliability.
After my small 1‑CPU, 2 GB server was compromised by a crypto‑mining virus that hijacked SSH access, I used VNC to investigate, identified malicious processes, traced infected files, removed cron jobs, restored system utilities, repaired SELinux, and closed the Redis vulnerability to fully recover the machine.
