A cloud engineer mistakenly set a disk snapshot to public, exposing a major client’s data, rushed a rollback, and then reflected on the root causes, highlighting the need for strict review, visual tools, and risk‑aware practices in high‑risk operations.
This article outlines the red‑blue adversarial testing process for a big‑data platform during the Double‑Eleven promotion, detailing its purpose, benefits, step‑by‑step execution, common issues, and recommendations to improve system reliability and security.
The article explains how a configurable yellow banner system can be used on web interfaces to promptly inform users about service disruptions, guide their actions, increase transparency, improve experience, and outline implementation considerations such as configurability, persistence, and independent deployment.
This article explains the concept and importance of MTTR (Mean Time To Repair), shows how to calculate it, and provides a comprehensive set of monitoring, alerting, rapid mitigation, tool‑assisted analysis, and team coordination techniques to significantly shorten incident resolution time and improve system reliability.
This article details Bilibili's data quality assurance framework, covering its evolution across four data platform stages, the architecture of its quality data warehouse, core capabilities such as a complete assurance system, digital‑driven continuous optimization, and efficient incident handling, plus case studies, future plans, and a Q&A session.
This article walks through Yuque’s October 23 service disruption, detailing each timeline milestone, analyzing the root causes, highlighting the importance of monitoring and data integrity checks, and offering concrete post‑mortem recommendations to improve future incident handling.
The article details Bilibili’s comprehensive alert‑management platform—its background, cloud‑vs‑self‑built solution comparison, closed‑loop design, distributed architecture, rule configuration, noise‑reduction, automated root‑cause analysis, and governance practices that cut weekly alerts from 1,000 to under 80, while outlining future enhancements.
Didi’s multi‑year data‑stability program for its customer‑service platform progressed through fault‑centered engineering, business‑aligned cross‑team work, and capability normalization, instituting pre‑, mid‑ and post‑fault safeguards, clear ownership, automated alerts and repair tools, which cut fault count by 42 % and more than doubled mean‑time‑to‑repair while boosting team communication and satisfaction.
This article details a real‑world Linux server intrusion, describing the observed symptoms, the forensic investigation using commands like ps, top, last, and grep, the removal of malicious cron jobs and backdoors, and the lessons learned for securing SSH, file attributes, and cloud security groups.
A self‑built Kubernetes cluster was compromised when an unprotected node with empty iptables and a kubelet that allowed anonymous API access was hijacked for Monero mining, prompting a detailed post‑mortem, root‑cause analysis, and hardening recommendations.
The article outlines fifteen essential cybersecurity metrics—thirteen process indicators such as mean detection and response times, and two result indicators like data loss incidents and security ROI—to help organizations evaluate, monitor, and improve their security posture and inform investment decisions.
The recent Xiaohongshu app crash sparked public outcry and a viral screenshot blaming a developer, but the article explains that software bugs are inevitable, responsibility lies with the whole team, and proper debugging, testing, and process improvements are the rational response.
The article outlines Tencent Search’s stability governance, detailing a multi‑layered availability architecture, disaster‑recovery mechanisms, precise monitoring, rapid emergency workflows, pre‑release interception, extensive automation, and a collaborative governance model that together enhance system resilience, incident detection, and swift remediation.
This article details a November 2022 database outage caused by a sudden CPU and load surge, explains how the team diagnosed the issue, outlines the emergency steps taken, and shares practical SQL performance optimization recommendations to prevent similar incidents.
This article details Huolala's journey in establishing a comprehensive technical stability framework, covering organizational challenges, risk governance, incident response, cultural initiatives, and future automation to enhance system reliability at scale.
Hello’s technical‑risk team created a comprehensive emergency‑response system for large‑scale promotions—prioritizing core scenarios, running high‑frequency drills, modeling fault‑portraits, defining metric‑based triggers and clear rollback actions—which delivered zero incidents during the 930 Big Sale, achieved over 80 % core‑line coverage, and now aims to automate plan selection and execution.
Robusta is a Python‑based open‑source platform that layers on top of monitoring stacks like Prometheus to automatically detect, diagnose, and remediate Kubernetes alerts through built‑in automations, optional web UI, and Helm‑based installation for cloud‑native environments.
This article details a recent server intrusion case, describing the observed symptoms, possible causes, step‑by‑step forensic investigation using commands like ps, top, grep and crontab, and comprehensive remediation actions such as tightening SSH security, unlocking and restoring system binaries, removing malicious scripts, and key lessons for future protection.
This article explains what stability assurance is, outlines a systematic workflow—including anomaly identification, monitoring configuration, impact assessment, and solution planning—and provides practical methods such as capacity estimation, traffic limiting, load testing, scaling, and pre‑heating to ensure services remain stable during both daily operations and high‑traffic events.
This guide explains essential incident response roles, how to quickly identify the source of a problem, prioritize actions, use efficient communication tools, and address human factors to improve your team's emergency response capabilities.
This article provides a step‑by‑step technical analysis of the DarkKomet remote‑access trojan, covering its capabilities, infection vectors, detection methods using TTP‑driven EDR, containment actions, eradication procedures, root‑cause forensics, and post‑incident recovery measures.
A senior engineer accidentally issued a DELETE request on an ElasticSearch index holding 17 million product records, triggering a massive data loss incident, and the team’s subsequent recovery strategies, scaling challenges, and process improvements are detailed to guide backend developers.
This article explains why nightly disk‑space alerts demand automated fault‑self‑healing, outlines the necessary process standards, describes monitoring platform dimensions, details a multi‑source self‑healing platform with CMDB integration, and provides practical options for script execution and result notification.
This article explains what a reliability culture is, why it matters, how to cultivate it through mission statements, early‑stage reliability testing, chaos‑engineering practices like GameDays and FireDrills, and how organizations can continuously learn from incidents to improve system availability and customer trust.
This article details a real‑world Linux server compromise, describing the symptoms, possible causes, investigative commands, hidden malicious scripts, file attribute locks, and practical remediation steps to restore the system and improve future security.
From kickoff meetings and traffic forecasting to load‑testing strategies, rate‑limiting designs, emergency runbooks, and post‑event retrospectives, this guide walks engineers through the complete technical workflow required to ensure a Double‑11‑scale e‑commerce promotion runs smoothly and safely.
This guide walks you through a complete Linux incident‑response workflow—identifying suspicious behavior, locating and terminating malicious processes, eliminating virus files, closing persistence mechanisms, and hardening the system to prevent future compromises—using practical shell commands and real‑world examples.
This post‑mortem details the July 2021 Bilibili outage caused by a Lua bug in the OpenResty‑based SLB, describing the timeline, root‑cause analysis, mitigation steps, and the technical and organizational improvements implemented to prevent similar incidents.
This article details the July 13, 2021 Bilibili service outage caused by a Lua‑based SLB CPU spike, describing the incident timeline, root‑cause analysis of a weight‑zero bug, mitigation steps including new SLB deployment, and the subsequent operational and architectural improvements.
On July 13 2021 Bilibili’s L7 SLB crashed when a recent Lua deployment set a balancer weight to the string “0”, producing a NaN value that triggered an infinite loop and 100 % CPU, prompting emergency restarts, a fresh cluster rollout, and long‑term safeguards such as automated provisioning, stricter Lua validation, and enhanced multi‑active disaster‑recovery processes.
This article details the challenges, design choices, deployment steps, detection model creation, data processing, visualization, and future plans of JD Daojia's security operations platform, highlighting the use of Graylog, Elasticsearch, and MongoDB to achieve scalable, real‑time threat detection and response.
The article examines why SRE teams experience overload due to high incident response demands, analyzes contributing factors such as production issues, alert volume, and manual processes, and proposes comprehensive mitigation steps including better testing, load management, and proactive error detection to reduce on‑call burden.
This article chronicles the discovery of a server breach used for cryptocurrency mining, detailing the malicious code, forensic analysis of the trojan's actions and artifacts, and the step‑by‑step remediation measures taken to secure the system.
This article chronicles the discovery of a server breach used for cryptocurrency mining, analyzes the malicious Python payload and its system modifications, and provides concrete remediation steps such as system reinstall, non‑root deployment, firewall hardening, and Nginx authentication.
This article explains how limited resources shape product requirement prioritization, test‑bug grading, product‑module classification, online bug severity, and incident response levels, offering practical frameworks and concrete grading tables to help teams make objective, value‑driven decisions throughout a product’s lifecycle.
To tackle growing production complexity and past incident delays, the consumer trading platform introduced a three‑tier NOC‑SLA with intelligent baselines powered by Facebook Prophet, streamlined alert rules, and an SOS‑linked workflow, boosting detection frequency, cutting critical response times to under five minutes, and improving overall system reliability while emphasizing ongoing baseline and rule maintenance.
Bilibili’s SRE team combines stability theory, detailed fault‑stage and operational metrics, and a unified emergency‑response platform—including on‑call scheduling, fault‑command incident commanders, automated fault portraits, and rapid post‑mortems—to transform frequent incidents into data‑driven, collaborative recoveries and lay groundwork for AI‑assisted self‑healing.
Wireless operations and maintenance (O&M) evolved from backend‑focused practices to address stability and performance of mobile‑device services, tackling low issue detection rates and delayed responses through improved monitoring, gray‑release tagging, phased rollouts, AI‑driven diagnostics, and automated release gates, while inviting collaborative development.
The article reviews ten major 2021 service outages—from Chinese platforms like Bilibili and Futu to global giants such as Facebook, Roblox, and AWS—analyzing their root causes, redundancy failures, and the operational lessons needed to prevent future black‑swans.
A Kubernetes node was compromised for Monero mining due to empty iptables, an exposed kubelet API, and a mis‑commented configuration, prompting a detailed post‑mortem and practical hardening steps to prevent similar attacks.
The article presents a comprehensive case study of KeMonitor, a one‑stop monitoring and alert platform built by 贝壳找房 to unify fragmented alerts, define lifecycle‑based governance, standardize alert metadata, implement graded subscription, on‑call escalation, silencing, self‑healing, and post‑mortem analysis, thereby improving incident response efficiency and reducing alert fatigue.
A Kubernetes cluster was compromised when a misconfigured kubelet allowed anonymous API access, enabling attackers to run a Monero miner; the post details the investigation, root causes, and practical hardening steps to prevent similar intrusions.
A comprehensive review of ten major 2021 internet outages—from domestic platforms like Bilibili and Futu to global services such as Facebook, Roblox, and AWS—examines their root causes, the role of infrastructure design, and the operational lessons needed to improve system resilience.
Huawei Cloud’s internal “blue‑team” launched over twenty coordinated attacks around Chinese New Year, but the company’s SRE “red‑team” and a dedicated 24/7 “special forces” unit detected, isolated, and resolved incidents within minutes, keeping failure rates below 0.01% and demonstrating advanced cloud operations and security practices.
A self‑built Kubernetes cluster suffered a crypto‑mining intrusion due to empty iptables and a misconfigured kubelet, prompting a detailed post‑mortem that outlines the symptoms, root‑cause analysis, and practical hardening steps to protect similar environments.
This guide walks through a four‑stage Linux incident‑response workflow—identifying symptoms, killing malicious processes, closing persistence mechanisms, and hardening the system—while providing the exact shell commands needed to detect and eradicate Linux malware.
This guide outlines the typical Linux and Windows environments encountered in security incidents, common threats such as mining and ransomware, and provides a step‑by‑step workflow with essential commands for process, user, network, and file investigation to identify and remediate compromises.
After discovering a suspicious process on one of our self‑built Kubernetes nodes, we traced the intrusion to a misconfigured kubelet that exposed the API, allowing attackers to run a Monero mining script, and we outline the investigation steps and hardening measures to prevent similar breaches.
In Xi'an's Lianhu District, a disgruntled former network administrator exploited self‑taught networking skills to illegally infiltrate a hospital's internal servers, remotely executing destructive operations that deleted critical files, disabled printers, CT and ultrasound machines, and ultimately caused the entire medical information system to collapse, prompting a police investigation that led to his arrest and criminal detention for damaging computer information systems.
This article examines real-world database deletion incidents, the associated legal consequences, and how DevOps culture and operational best‑practices can turn such mistakes into learning opportunities rather than career‑ending failures.
Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.
This article details how a major securities firm analyzed business stability, built a comprehensive stability engineering platform using chaos engineering, practiced extensive fault‑injection drills, and outlines future directions such as random‑scenario exercises, red‑blue battles, and AI‑driven risk detection.
This guide explains how to design and practice a structured incident‑response process—defining problems, applying quick‑recovery steps, analyzing root causes, standardizing solutions, and assigning critical roles—to dramatically reduce production outage duration.
This guide outlines a structured approach to incident response, detailing problem definition, temporary fixes, root‑cause analysis, solution design, implementation, and standardization, while highlighting four critical roles—commander, communicator, rapid‑recovery lead, and diagnosis lead—to ensure swift, coordinated recovery of production services.
This guide walks through common Linux emergency scenarios—such as mining malware, ransomware, and backdoors—detailing a step‑by‑step workflow and providing essential command‑line tools for process, user, network, and file investigation on CentOS 6 and Windows Server 2008 systems.
When a server suddenly generated 800 MB of outbound traffic and SSH became unresponsive, the author traced the issue to a hidden backdoor, blocked the malicious IP, identified compromised binaries, removed malicious processes and startup scripts, and outlined preventive security measures.
This article presents a comprehensive, step‑by‑step framework for guaranteeing system reliability during high‑traffic promotional periods, covering SRE hierarchy, stability criteria, profiling, monitoring, capacity planning, incident response, and post‑event analysis to help teams build resilient services.
This article recounts a production server data loss caused by an erroneous rm -rf command, describes how ext3grep and extundelete were used to attempt file restoration, and explains how MySQL binlog replay finally recovered critical data while sharing lessons learned for future incident handling.
This article shares a step‑by‑step Linux incident‑response workflow, including an automated Bash information‑gathering script, analysis of malicious cron jobs and a 439‑line mining malware, its SSH‑based lateral spread, and practical cleanup procedures with a reusable toolbox on GitHub.
This article walks through a real‑world Linux mining malware infection, detailing how the attacker hid a malicious cron job, used LD_PRELOAD rootkits, propagated via SSH keys, and how the analyst uncovered and removed the threat using busybox, strace, and careful forensic commands.
iQIYI’s SOAR platform, built on StackStorm and the Walkoff visual editor, integrates security components, scripts, chat‑ops bots, and a mini‑program to automate detection and response, cutting MTTR by roughly 75% across high‑frequency routine tasks and low‑frequency critical incidents while planning broader coverage and knowledge‑base expansion.
This article examines the critical role of stability governance in evolving systems, outlines a three‑stage framework—usability, monitoring alerts, and online emergency—illustrated with a case study of an electronic waybill service, and shares concrete strategies for prevention, detection, response, and post‑mortem to achieve predictable, observable, and fast‑acting reliability.
This article analyzes the concept of stability governance, outlines its five fault‑management sub‑domains, examines the pain points of an electronic waybill service, and presents a comprehensive three‑phase strategy—prevention, perception, reach, mitigation, and post‑mortem—backed by concrete implementation steps in availability, monitoring, and online emergency handling.
This article explains how Netflix built the Telltale monitoring system to consolidate data sources, provide multidimensional health assessments, deliver intelligent alerts, and streamline incident management for over 100 production applications, reducing on‑call fatigue and improving service reliability.
After my small 1‑CPU, 2 GB server was compromised by a crypto‑mining virus that hijacked SSH access, I used VNC to investigate, identified malicious processes, traced infected files, removed cron jobs, restored system utilities, repaired SELinux, and closed the Redis vulnerability to fully recover the machine.
This article explains how the JSTracker platform was used to build a comprehensive end‑to‑end front‑end monitoring and data analysis solution that meets the 1‑5‑10 safety production goal—detecting issues within one minute, locating them in five, and fixing them in ten—by improving coverage, subscription, metrics, and gray‑release monitoring for Alibaba’s Double‑11 promotion.
This article explains how JD Cloud builds a multi‑layered, end‑to‑end security architecture—including five defense layers, four implementation stages, three guiding principles, and two key focuses—to protect high‑traffic e‑commerce events such as 618 and 11.11 from attacks and ensure stable, safe operations.
This article explains practical methods for detecting account security threats—such as blacklisted, expired, or abnormal login behaviors—by analyzing Linux and Windows login logs, defining detection rules, and leveraging automated tools to generate timely alerts and reduce security risks.
A sudden P1 incident reset all user passwords, and after a thorough investigation the team discovered that a security‑scanning tool’s weak‑password check repeatedly hit login attempts, triggering a bug that caused the outage, highlighting the critical need for proper incident response and security engineering.
During the Mid‑Autumn Festival I received a seemingly harmless mooncake email, suspected it was a phishing test, and then used a virtual machine, network‑capture tools, Shodan, and open‑source intelligence to trace the malicious link back to its source and exposed the underlying infrastructure.
A Shanghai court sentenced programmer He Mou to six years in prison for deliberately deleting all SaaS data of Weimeng, causing an eight‑day outage, over 300 million yuan in losses, and prompting a discussion on on‑premise versus cloud data protection strategies.
This article explains how enterprises adopting SRE can design a comprehensive observability platform—covering metrics, logs, and tracing—while also detailing effective incident response, post‑mortem practices, testing, capacity planning, automation tool development, and user‑experience focus to improve overall operational reliability.
The article analyzes the 2012 Knight Capital Group disaster, showing how a manual deployment error, lingering legacy code, missing kill‑switch, and inadequate monitoring caused a $4.6 billion loss within 45 minutes, and extracts key DevOps best‑practice lessons to prevent similar failures.
The interview with Sun Bin, head of Qunar's website operations, explains how the team acted as a specialized "BlackOps" unit to provide robust technical guarantees, automate problem‑solving, protect user data, optimize resources, and maintain continuous service during the COVID‑19 outbreak.
This article demonstrates how to audit MySQL user actions by configuring init_connect, creating an audit log table, enabling binlog, and analyzing binlog entries to identify the user and IP responsible for accidental table deletions.
The article examines how personal grievances have led to tragic events—from a bus driver’s fatal crash to destructive IT incidents—highlighting the need for psychological care, robust security policies, and a DevSecOps mindset to prevent such revenge‑driven catastrophes.
During the pandemic’s “停课不停学” surge, Tencent Classroom tackled a 120‑fold traffic jump by rapidly deploying Grafana dashboards, Kibana logs, internal Moniter and cloud monitoring tools, establishing a three‑layer feedback‑alert‑on‑call model, and now plans automation, unified visualizations, and chaos‑engineering to further boost observability and service reliability.
The Zhengda Hospital HIS database outage, caused by unauthorized scripts and poor permission controls, sparked a detailed discussion on how to prevent reckless production testing, enforce proper authorization, design efficient yet secure workflows, improve outsourcing oversight, and build robust emergency and compliance practices.
This article details a real‑world Linux rootkit intrusion, describing the symptoms, forensic analysis techniques, evidence uncovered, the underlying Awstats vulnerability, and a comprehensive remediation plan to secure the server and prevent future compromises.
On Feb 23, Weimeng suffered a large‑scale system outage caused by a core operations staff mistakenly deleting production databases, prompting a multi‑day recovery effort with Tencent Cloud support; the article examines the incident’s background, historical parallels, crisis response, and broader operational insights for DevOps and reliability engineering.
The article analyzes the recent Weimeng data‑loss incident, explains why recovery took 36 hours, highlights insider abuse, and offers a practical guide for small and large teams covering reliable backups, minimal‑privilege management, and cloud‑based disaster‑recovery solutions.
After a core operations staff accidentally deleted Weimeng’s production database in February, the platform endured a multi‑day outage, prompting a transparent crisis response, extensive Tencent Cloud support, and a deep analysis of recovery challenges, operational best practices, and the broader lessons for modern DevOps teams.
The article analyzes the recent Weimeng database deletion incident, explains why recovery took 36 hours, and provides practical guidance on backup practices, minimal‑privilege management, and cloud‑based disaster recovery to prevent similar data loss in small and large organizations.
A malicious insider deleted Weimob's primary and backup databases, prompting a slow recovery effort and highlighting the critical need for stricter permission controls and reliable backup mechanisms to prevent similar incidents.
On February 25, Weimob disclosed that a core operations employee maliciously destroyed SaaS business data, prompting police involvement and a rapid recovery effort, and the incident underscores the need for comprehensive backup, cloud redundancy, strict access controls, automated deployment, and proactive risk planning.
This article outlines how Ping An’s IT operations team systematically prepares for high‑traffic business events—detailing service assessment, architecture mapping, configuration audits, monitoring design, capacity planning, stress testing, and coordinated incident response—to guarantee reliability and performance under massive concurrent loads.
During the 2020 Chinese New Year lockdown, a travel platform mobilized its development, product, and operations teams to rapidly build refund systems, coordinate with suppliers, and ensure continuous online services, showcasing a user‑first, cross‑functional emergency strategy that balanced technical delivery with intense customer pressure.
This article details a step‑by‑step investigation of repeated follower process alerts in a Paxos‑based distributed coordination service, revealing a Java GC pause‑induced memory leak in the front‑end Proxy and describing the rapid mitigation actions taken to restore system stability.
This guide walks you through a complete Linux incident‑response workflow—identifying suspicious behavior, terminating malicious processes, eradicating virus files, closing persistence mechanisms, and hardening the system—while providing concrete shell commands and practical tips for each stage.
This guide walks you through a complete Linux emergency response workflow—identifying suspicious behavior, terminating malicious processes, removing infected files, eliminating persistence mechanisms, hardening the system, and adding command auditing—using practical shell commands and examples.
This article examines how Google SRE limits weekly alerts to ten, compares it with typical Chinese internet operations teams, and provides practical strategies—including on‑call scheduling, alert escalation, automation, dashboard optimization, and team management—to dramatically reduce alert volume and improve incident response.
The article shares practical Ops insights, using a simple RAID change incident to illustrate why operations teams must understand change background, choose optimal timing, act as project managers, and follow a structured change process to protect production environments.
This article shares a set of operational Murphy’s laws, practical process‑management tips, and automation strategies to help ops engineers reduce human error, improve safety, stability, efficiency, and cost‑saving in daily work.
A detailed forensic walk‑through reveals how a compromised Alibaba Cloud server was seized via a weak root password, a disguised gpg-agentd binary, malicious cron jobs, and Redis configuration abuse, ultimately enabling password‑less SSH access and large‑scale network scanning for cryptocurrency mining.
This article details a multi‑stage, file‑less attack that leveraged weak SQL Server credentials, Transact‑SQL stored procedures, and WMI to download and execute a downloader (cabs.exe) which fetched multiple botnet components, and explains the forensic steps and remediation measures taken to eradicate the threat.
This article explains how a bank can create a specialized Operations SWAT team, define its role, adopt seven essential “weapons” such as layered monitoring, intelligent alerts, communication protocols, automation, and disaster‑recovery tactics, and continuously train the team to meet strict five‑minute recovery targets.
This article explains how a bank can create a specialized SWAT‑style operations team, define its roles, adopt seven essential "weapons" such as monitoring and intelligent alerts, and apply ten tactical processes—from communication to automation—to meet strict five‑minute recovery and regulatory requirements.
This article details a multi‑stage server compromise that injected gambling pages, planted hidden accounts, deployed crypto‑mining software, and opened unnecessary ports, providing step‑by‑step forensic analysis, code inspection, emergency response actions, and indicators of compromise.
This article explains the concept of operations security, why it has become critical, enumerates common mis‑configurations and vulnerabilities such as open ports, weak permissions, insecure scripts and supply‑chain risks, and provides a comprehensive set of best‑practice guidelines and an enterprise‑level framework to build a resilient operations security posture.
This article recounts a Redis server hijack incident, detailing the detection, forensic investigation, removal of malicious files and cron jobs, and provides practical hardening recommendations to prevent similar attacks on Linux environments.
