This article provides a step‑by‑step technical analysis of the DarkKomet remote‑access trojan, covering its capabilities, infection vectors, detection methods using TTP‑driven EDR, containment actions, eradication procedures, root‑cause forensics, and post‑incident recovery measures.
A senior engineer accidentally issued a DELETE request on an ElasticSearch index holding 17 million product records, triggering a massive data loss incident, and the team’s subsequent recovery strategies, scaling challenges, and process improvements are detailed to guide backend developers.
This article explains why nightly disk‑space alerts demand automated fault‑self‑healing, outlines the necessary process standards, describes monitoring platform dimensions, details a multi‑source self‑healing platform with CMDB integration, and provides practical options for script execution and result notification.
This article explains what a reliability culture is, why it matters, how to cultivate it through mission statements, early‑stage reliability testing, chaos‑engineering practices like GameDays and FireDrills, and how organizations can continuously learn from incidents to improve system availability and customer trust.
This article details a real‑world Linux server compromise, describing the symptoms, possible causes, investigative commands, hidden malicious scripts, file attribute locks, and practical remediation steps to restore the system and improve future security.
From kickoff meetings and traffic forecasting to load‑testing strategies, rate‑limiting designs, emergency runbooks, and post‑event retrospectives, this guide walks engineers through the complete technical workflow required to ensure a Double‑11‑scale e‑commerce promotion runs smoothly and safely.
This guide walks you through a complete Linux incident‑response workflow—identifying suspicious behavior, locating and terminating malicious processes, eliminating virus files, closing persistence mechanisms, and hardening the system to prevent future compromises—using practical shell commands and real‑world examples.
This post‑mortem details the July 2021 Bilibili outage caused by a Lua bug in the OpenResty‑based SLB, describing the timeline, root‑cause analysis, mitigation steps, and the technical and organizational improvements implemented to prevent similar incidents.
This article details the July 13, 2021 Bilibili service outage caused by a Lua‑based SLB CPU spike, describing the incident timeline, root‑cause analysis of a weight‑zero bug, mitigation steps including new SLB deployment, and the subsequent operational and architectural improvements.
On July 13 2021 Bilibili’s L7 SLB crashed when a recent Lua deployment set a balancer weight to the string “0”, producing a NaN value that triggered an infinite loop and 100 % CPU, prompting emergency restarts, a fresh cluster rollout, and long‑term safeguards such as automated provisioning, stricter Lua validation, and enhanced multi‑active disaster‑recovery processes.
This article details the challenges, design choices, deployment steps, detection model creation, data processing, visualization, and future plans of JD Daojia's security operations platform, highlighting the use of Graylog, Elasticsearch, and MongoDB to achieve scalable, real‑time threat detection and response.
The article examines why SRE teams experience overload due to high incident response demands, analyzes contributing factors such as production issues, alert volume, and manual processes, and proposes comprehensive mitigation steps including better testing, load management, and proactive error detection to reduce on‑call burden.
This article chronicles the discovery of a server breach used for cryptocurrency mining, detailing the malicious code, forensic analysis of the trojan's actions and artifacts, and the step‑by‑step remediation measures taken to secure the system.
This article chronicles the discovery of a server breach used for cryptocurrency mining, analyzes the malicious Python payload and its system modifications, and provides concrete remediation steps such as system reinstall, non‑root deployment, firewall hardening, and Nginx authentication.
This article explains how limited resources shape product requirement prioritization, test‑bug grading, product‑module classification, online bug severity, and incident response levels, offering practical frameworks and concrete grading tables to help teams make objective, value‑driven decisions throughout a product’s lifecycle.
To tackle growing production complexity and past incident delays, the consumer trading platform introduced a three‑tier NOC‑SLA with intelligent baselines powered by Facebook Prophet, streamlined alert rules, and an SOS‑linked workflow, boosting detection frequency, cutting critical response times to under five minutes, and improving overall system reliability while emphasizing ongoing baseline and rule maintenance.
Bilibili’s SRE team combines stability theory, detailed fault‑stage and operational metrics, and a unified emergency‑response platform—including on‑call scheduling, fault‑command incident commanders, automated fault portraits, and rapid post‑mortems—to transform frequent incidents into data‑driven, collaborative recoveries and lay groundwork for AI‑assisted self‑healing.
Wireless operations and maintenance (O&M) evolved from backend‑focused practices to address stability and performance of mobile‑device services, tackling low issue detection rates and delayed responses through improved monitoring, gray‑release tagging, phased rollouts, AI‑driven diagnostics, and automated release gates, while inviting collaborative development.
The article reviews ten major 2021 service outages—from Chinese platforms like Bilibili and Futu to global giants such as Facebook, Roblox, and AWS—analyzing their root causes, redundancy failures, and the operational lessons needed to prevent future black‑swans.
A Kubernetes node was compromised for Monero mining due to empty iptables, an exposed kubelet API, and a mis‑commented configuration, prompting a detailed post‑mortem and practical hardening steps to prevent similar attacks.
The article presents a comprehensive case study of KeMonitor, a one‑stop monitoring and alert platform built by 贝壳找房 to unify fragmented alerts, define lifecycle‑based governance, standardize alert metadata, implement graded subscription, on‑call escalation, silencing, self‑healing, and post‑mortem analysis, thereby improving incident response efficiency and reducing alert fatigue.
A Kubernetes cluster was compromised when a misconfigured kubelet allowed anonymous API access, enabling attackers to run a Monero miner; the post details the investigation, root causes, and practical hardening steps to prevent similar intrusions.
A comprehensive review of ten major 2021 internet outages—from domestic platforms like Bilibili and Futu to global services such as Facebook, Roblox, and AWS—examines their root causes, the role of infrastructure design, and the operational lessons needed to improve system resilience.
Huawei Cloud’s internal “blue‑team” launched over twenty coordinated attacks around Chinese New Year, but the company’s SRE “red‑team” and a dedicated 24/7 “special forces” unit detected, isolated, and resolved incidents within minutes, keeping failure rates below 0.01% and demonstrating advanced cloud operations and security practices.
A self‑built Kubernetes cluster suffered a crypto‑mining intrusion due to empty iptables and a misconfigured kubelet, prompting a detailed post‑mortem that outlines the symptoms, root‑cause analysis, and practical hardening steps to protect similar environments.
This guide walks through a four‑stage Linux incident‑response workflow—identifying symptoms, killing malicious processes, closing persistence mechanisms, and hardening the system—while providing the exact shell commands needed to detect and eradicate Linux malware.
This guide outlines the typical Linux and Windows environments encountered in security incidents, common threats such as mining and ransomware, and provides a step‑by‑step workflow with essential commands for process, user, network, and file investigation to identify and remediate compromises.
After discovering a suspicious process on one of our self‑built Kubernetes nodes, we traced the intrusion to a misconfigured kubelet that exposed the API, allowing attackers to run a Monero mining script, and we outline the investigation steps and hardening measures to prevent similar breaches.
In Xi'an's Lianhu District, a disgruntled former network administrator exploited self‑taught networking skills to illegally infiltrate a hospital's internal servers, remotely executing destructive operations that deleted critical files, disabled printers, CT and ultrasound machines, and ultimately caused the entire medical information system to collapse, prompting a police investigation that led to his arrest and criminal detention for damaging computer information systems.
This article examines real-world database deletion incidents, the associated legal consequences, and how DevOps culture and operational best‑practices can turn such mistakes into learning opportunities rather than career‑ending failures.
Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.
This article details how a major securities firm analyzed business stability, built a comprehensive stability engineering platform using chaos engineering, practiced extensive fault‑injection drills, and outlines future directions such as random‑scenario exercises, red‑blue battles, and AI‑driven risk detection.
This guide explains how to design and practice a structured incident‑response process—defining problems, applying quick‑recovery steps, analyzing root causes, standardizing solutions, and assigning critical roles—to dramatically reduce production outage duration.
This guide outlines a structured approach to incident response, detailing problem definition, temporary fixes, root‑cause analysis, solution design, implementation, and standardization, while highlighting four critical roles—commander, communicator, rapid‑recovery lead, and diagnosis lead—to ensure swift, coordinated recovery of production services.
This guide walks through common Linux emergency scenarios—such as mining malware, ransomware, and backdoors—detailing a step‑by‑step workflow and providing essential command‑line tools for process, user, network, and file investigation on CentOS 6 and Windows Server 2008 systems.
When a server suddenly generated 800 MB of outbound traffic and SSH became unresponsive, the author traced the issue to a hidden backdoor, blocked the malicious IP, identified compromised binaries, removed malicious processes and startup scripts, and outlined preventive security measures.
This article presents a comprehensive, step‑by‑step framework for guaranteeing system reliability during high‑traffic promotional periods, covering SRE hierarchy, stability criteria, profiling, monitoring, capacity planning, incident response, and post‑event analysis to help teams build resilient services.
This article recounts a production server data loss caused by an erroneous rm -rf command, describes how ext3grep and extundelete were used to attempt file restoration, and explains how MySQL binlog replay finally recovered critical data while sharing lessons learned for future incident handling.
This article shares a step‑by‑step Linux incident‑response workflow, including an automated Bash information‑gathering script, analysis of malicious cron jobs and a 439‑line mining malware, its SSH‑based lateral spread, and practical cleanup procedures with a reusable toolbox on GitHub.
This article walks through a real‑world Linux mining malware infection, detailing how the attacker hid a malicious cron job, used LD_PRELOAD rootkits, propagated via SSH keys, and how the analyst uncovered and removed the threat using busybox, strace, and careful forensic commands.
iQIYI’s SOAR platform, built on StackStorm and the Walkoff visual editor, integrates security components, scripts, chat‑ops bots, and a mini‑program to automate detection and response, cutting MTTR by roughly 75% across high‑frequency routine tasks and low‑frequency critical incidents while planning broader coverage and knowledge‑base expansion.
This article examines the critical role of stability governance in evolving systems, outlines a three‑stage framework—usability, monitoring alerts, and online emergency—illustrated with a case study of an electronic waybill service, and shares concrete strategies for prevention, detection, response, and post‑mortem to achieve predictable, observable, and fast‑acting reliability.
This article analyzes the concept of stability governance, outlines its five fault‑management sub‑domains, examines the pain points of an electronic waybill service, and presents a comprehensive three‑phase strategy—prevention, perception, reach, mitigation, and post‑mortem—backed by concrete implementation steps in availability, monitoring, and online emergency handling.
This article explains how Netflix built the Telltale monitoring system to consolidate data sources, provide multidimensional health assessments, deliver intelligent alerts, and streamline incident management for over 100 production applications, reducing on‑call fatigue and improving service reliability.
After my small 1‑CPU, 2 GB server was compromised by a crypto‑mining virus that hijacked SSH access, I used VNC to investigate, identified malicious processes, traced infected files, removed cron jobs, restored system utilities, repaired SELinux, and closed the Redis vulnerability to fully recover the machine.
This article explains how the JSTracker platform was used to build a comprehensive end‑to‑end front‑end monitoring and data analysis solution that meets the 1‑5‑10 safety production goal—detecting issues within one minute, locating them in five, and fixing them in ten—by improving coverage, subscription, metrics, and gray‑release monitoring for Alibaba’s Double‑11 promotion.
This article explains how JD Cloud builds a multi‑layered, end‑to‑end security architecture—including five defense layers, four implementation stages, three guiding principles, and two key focuses—to protect high‑traffic e‑commerce events such as 618 and 11.11 from attacks and ensure stable, safe operations.
This article explains practical methods for detecting account security threats—such as blacklisted, expired, or abnormal login behaviors—by analyzing Linux and Windows login logs, defining detection rules, and leveraging automated tools to generate timely alerts and reduce security risks.
A sudden P1 incident reset all user passwords, and after a thorough investigation the team discovered that a security‑scanning tool’s weak‑password check repeatedly hit login attempts, triggering a bug that caused the outage, highlighting the critical need for proper incident response and security engineering.
During the Mid‑Autumn Festival I received a seemingly harmless mooncake email, suspected it was a phishing test, and then used a virtual machine, network‑capture tools, Shodan, and open‑source intelligence to trace the malicious link back to its source and exposed the underlying infrastructure.
A Shanghai court sentenced programmer He Mou to six years in prison for deliberately deleting all SaaS data of Weimeng, causing an eight‑day outage, over 300 million yuan in losses, and prompting a discussion on on‑premise versus cloud data protection strategies.
This article explains how enterprises adopting SRE can design a comprehensive observability platform—covering metrics, logs, and tracing—while also detailing effective incident response, post‑mortem practices, testing, capacity planning, automation tool development, and user‑experience focus to improve overall operational reliability.
The article analyzes the 2012 Knight Capital Group disaster, showing how a manual deployment error, lingering legacy code, missing kill‑switch, and inadequate monitoring caused a $4.6 billion loss within 45 minutes, and extracts key DevOps best‑practice lessons to prevent similar failures.
The interview with Sun Bin, head of Qunar's website operations, explains how the team acted as a specialized "BlackOps" unit to provide robust technical guarantees, automate problem‑solving, protect user data, optimize resources, and maintain continuous service during the COVID‑19 outbreak.
This article demonstrates how to audit MySQL user actions by configuring init_connect, creating an audit log table, enabling binlog, and analyzing binlog entries to identify the user and IP responsible for accidental table deletions.
The article examines how personal grievances have led to tragic events—from a bus driver’s fatal crash to destructive IT incidents—highlighting the need for psychological care, robust security policies, and a DevSecOps mindset to prevent such revenge‑driven catastrophes.
During the pandemic’s “停课不停学” surge, Tencent Classroom tackled a 120‑fold traffic jump by rapidly deploying Grafana dashboards, Kibana logs, internal Moniter and cloud monitoring tools, establishing a three‑layer feedback‑alert‑on‑call model, and now plans automation, unified visualizations, and chaos‑engineering to further boost observability and service reliability.
The Zhengda Hospital HIS database outage, caused by unauthorized scripts and poor permission controls, sparked a detailed discussion on how to prevent reckless production testing, enforce proper authorization, design efficient yet secure workflows, improve outsourcing oversight, and build robust emergency and compliance practices.
This article details a real‑world Linux rootkit intrusion, describing the symptoms, forensic analysis techniques, evidence uncovered, the underlying Awstats vulnerability, and a comprehensive remediation plan to secure the server and prevent future compromises.
On Feb 23, Weimeng suffered a large‑scale system outage caused by a core operations staff mistakenly deleting production databases, prompting a multi‑day recovery effort with Tencent Cloud support; the article examines the incident’s background, historical parallels, crisis response, and broader operational insights for DevOps and reliability engineering.
The article analyzes the recent Weimeng data‑loss incident, explains why recovery took 36 hours, highlights insider abuse, and offers a practical guide for small and large teams covering reliable backups, minimal‑privilege management, and cloud‑based disaster‑recovery solutions.
After a core operations staff accidentally deleted Weimeng’s production database in February, the platform endured a multi‑day outage, prompting a transparent crisis response, extensive Tencent Cloud support, and a deep analysis of recovery challenges, operational best practices, and the broader lessons for modern DevOps teams.
The article analyzes the recent Weimeng database deletion incident, explains why recovery took 36 hours, and provides practical guidance on backup practices, minimal‑privilege management, and cloud‑based disaster recovery to prevent similar data loss in small and large organizations.
A malicious insider deleted Weimob's primary and backup databases, prompting a slow recovery effort and highlighting the critical need for stricter permission controls and reliable backup mechanisms to prevent similar incidents.
On February 25, Weimob disclosed that a core operations employee maliciously destroyed SaaS business data, prompting police involvement and a rapid recovery effort, and the incident underscores the need for comprehensive backup, cloud redundancy, strict access controls, automated deployment, and proactive risk planning.
This article outlines how Ping An’s IT operations team systematically prepares for high‑traffic business events—detailing service assessment, architecture mapping, configuration audits, monitoring design, capacity planning, stress testing, and coordinated incident response—to guarantee reliability and performance under massive concurrent loads.
During the 2020 Chinese New Year lockdown, a travel platform mobilized its development, product, and operations teams to rapidly build refund systems, coordinate with suppliers, and ensure continuous online services, showcasing a user‑first, cross‑functional emergency strategy that balanced technical delivery with intense customer pressure.
This article details a step‑by‑step investigation of repeated follower process alerts in a Paxos‑based distributed coordination service, revealing a Java GC pause‑induced memory leak in the front‑end Proxy and describing the rapid mitigation actions taken to restore system stability.
This guide walks you through a complete Linux incident‑response workflow—identifying suspicious behavior, terminating malicious processes, eradicating virus files, closing persistence mechanisms, and hardening the system—while providing concrete shell commands and practical tips for each stage.
This guide walks you through a complete Linux emergency response workflow—identifying suspicious behavior, terminating malicious processes, removing infected files, eliminating persistence mechanisms, hardening the system, and adding command auditing—using practical shell commands and examples.
This article examines how Google SRE limits weekly alerts to ten, compares it with typical Chinese internet operations teams, and provides practical strategies—including on‑call scheduling, alert escalation, automation, dashboard optimization, and team management—to dramatically reduce alert volume and improve incident response.
The article shares practical Ops insights, using a simple RAID change incident to illustrate why operations teams must understand change background, choose optimal timing, act as project managers, and follow a structured change process to protect production environments.
This article shares a set of operational Murphy’s laws, practical process‑management tips, and automation strategies to help ops engineers reduce human error, improve safety, stability, efficiency, and cost‑saving in daily work.
A detailed forensic walk‑through reveals how a compromised Alibaba Cloud server was seized via a weak root password, a disguised gpg-agentd binary, malicious cron jobs, and Redis configuration abuse, ultimately enabling password‑less SSH access and large‑scale network scanning for cryptocurrency mining.
This article details a multi‑stage, file‑less attack that leveraged weak SQL Server credentials, Transact‑SQL stored procedures, and WMI to download and execute a downloader (cabs.exe) which fetched multiple botnet components, and explains the forensic steps and remediation measures taken to eradicate the threat.
This article explains how a bank can create a specialized Operations SWAT team, define its role, adopt seven essential “weapons” such as layered monitoring, intelligent alerts, communication protocols, automation, and disaster‑recovery tactics, and continuously train the team to meet strict five‑minute recovery targets.
This article explains how a bank can create a specialized SWAT‑style operations team, define its roles, adopt seven essential "weapons" such as monitoring and intelligent alerts, and apply ten tactical processes—from communication to automation—to meet strict five‑minute recovery and regulatory requirements.
This article details a multi‑stage server compromise that injected gambling pages, planted hidden accounts, deployed crypto‑mining software, and opened unnecessary ports, providing step‑by‑step forensic analysis, code inspection, emergency response actions, and indicators of compromise.
This article explains the concept of operations security, why it has become critical, enumerates common mis‑configurations and vulnerabilities such as open ports, weak permissions, insecure scripts and supply‑chain risks, and provides a comprehensive set of best‑practice guidelines and an enterprise‑level framework to build a resilient operations security posture.
This article recounts a Redis server hijack incident, detailing the detection, forensic investigation, removal of malicious files and cron jobs, and provides practical hardening recommendations to prevent similar attacks on Linux environments.
Effective intrusion detection for large enterprises hinges on combining signature‑based pattern matching with baseline anomaly modeling, gathering comprehensive host and network logs, focusing on the GetShell foothold, managing alert fatigue, and integrating AI‑enhanced feature engineering while maintaining robust operational foundations and continuous expertise development.
This article presents a comprehensive guide to operational security, covering essential habits, a layered technical architecture, access‑control strategies, CI/CD safeguards, DDoS mitigation, data protection, incident‑response procedures, and collaboration with IT, security, and network teams.
This article details a step‑by‑step incident‑response case study of a Windows internal‑network Trojan that exploited SMB port 445, describing how alerts were identified, malicious processes were traced, terminated, and fully removed using tools such as netstat, PChunter, and process monitoring utilities.
This guide outlines a step‑by‑step approach for taking over new operational responsibilities, covering communication with development leaders, business overview, asset inventory, basic and business‑specific monitoring, standardization, SOP creation, failure drills, cost and capacity planning, and effective cross‑team communication.
This guide outlines a step‑by‑step operations playbook for assuming responsibility of a new business service, covering initial communication, asset inventory, monitoring setup, standardization, SOP creation, incident drills, ongoing optimization, and effective cross‑team communication to ensure stable, low‑cost, and high‑quality service delivery.
This article analyzes the recent surge of Redis unauthenticated attacks that install a worm, use pnscan for lateral scanning, modify system settings, and launch cryptocurrency mining, while providing detailed script breakdowns and remediation steps.
This article details a real-world incident response to a compromised website that redirected visitors to a gambling site, covering intrusion analysis, server remediation, log tracing, and comprehensive cleanup steps to restore security.
A seasoned operations engineer shares a comprehensive guide covering online operation standards, data handling, security hardening, daily monitoring, performance tuning, and the right mindset to prevent costly incidents and ensure stable, secure, and efficient production environments.
A production server running Tomcat, MySQL, MongoDB and ActiveMQ was taken down by a malicious cron job that executed a cryptomining script, and the article walks through the investigation, removal, and hardening steps to fully recover and secure the system.
The 2017 Double 11 stability monitoring project introduced a four‑layer monitoring architecture—including customer & sentiment, business, system water‑level, and infrastructure monitoring—along with data archiving and system‑level reliability measures to detect, respond to, and mitigate issues far faster than traditional manual processes.
A DBA on call discovers a sudden traffic spike caused by a few massive hot keys on a storage server, quickly isolates the issue, migrates data, applies throttling and caching, and outlines automation ideas to prevent future overloads, illustrating practical database operations and incident response.
This article shares practical methods for frontline engineers to quickly understand, assess, and resolve online system problems by categorizing system layers, evaluating impact, using essential Linux monitoring tools, and applying systematic troubleshooting and design‑for‑failure strategies to minimize downtime.
This article presents a systematic methodology for diagnosing and resolving online system issues, covering system understanding, impact assessment, rapid recovery techniques, detailed troubleshooting steps with Linux and Java tools, and design principles to mitigate future failures.
After a sudden server crash, the author traced a ransomware note, uncovered a Bash Shellshock exploit through log analysis and crafted GET requests, verified the vulnerability, upgraded Bash, and applied post‑compromise hardening steps to fully recover the system.
After a mistaken rm -rf command wiped an entire production server, I detail the step‑by‑step recovery using ext3grep, a custom restoration script, extundelete attempts, and finally MySQL binlog replay, highlighting the challenges and lessons learned.
This guide outlines comprehensive Linux security practices for administrators, covering account and login protection, service minimization, password and key authentication, sudo usage, system welcome message hardening, remote access safeguards, filesystem permissions, rootkit detection tools, and step‑by‑step response procedures after a server compromise.
After a mistaken rm -rf command wiped an entire production server—including MySQL data—the author chronicles a step‑by‑step recovery using ext3grep, custom scripts, and binlog restoration, highlighting lessons learned and best practices for future incident handling.
This article recounts a Ctrip security engineer’s evolution from early Unix‑based operations to fully automated network security, highlighting challenges in forecasting, application security integration, rapid incident response, and large‑scale firewall automation within a fast‑growing enterprise.
This article recounts a recent incident where a Struts2 vulnerability was exploited to run mining malware, detailing the discovery process, forensic analysis of services, processes, network listeners, and the step‑by‑step remediation measures including script‑based scans, permission hardening, and upgrading Struts2.
This article outlines a comprehensive PDCA‑based methodology for e‑commerce platforms to proactively prevent issues, quickly detect anomalies, and execute rapid decisions during large‑scale promotions, covering system goal definition, performance evaluation, capacity planning, SLA management, and team/process maturity.
