This article presents a systematic methodology for diagnosing and resolving online system issues, covering system understanding, impact assessment, rapid recovery techniques, detailed troubleshooting steps with Linux and Java tools, and design principles to mitigate future failures.
After a sudden server crash, the author traced a ransomware note, uncovered a Bash Shellshock exploit through log analysis and crafted GET requests, verified the vulnerability, upgraded Bash, and applied post‑compromise hardening steps to fully recover the system.
After a mistaken rm -rf command wiped an entire production server, I detail the step‑by‑step recovery using ext3grep, a custom restoration script, extundelete attempts, and finally MySQL binlog replay, highlighting the challenges and lessons learned.
This guide outlines comprehensive Linux security practices for administrators, covering account and login protection, service minimization, password and key authentication, sudo usage, system welcome message hardening, remote access safeguards, filesystem permissions, rootkit detection tools, and step‑by‑step response procedures after a server compromise.
After a mistaken rm -rf command wiped an entire production server—including MySQL data—the author chronicles a step‑by‑step recovery using ext3grep, custom scripts, and binlog restoration, highlighting lessons learned and best practices for future incident handling.
This article recounts a Ctrip security engineer’s evolution from early Unix‑based operations to fully automated network security, highlighting challenges in forecasting, application security integration, rapid incident response, and large‑scale firewall automation within a fast‑growing enterprise.
This article recounts a recent incident where a Struts2 vulnerability was exploited to run mining malware, detailing the discovery process, forensic analysis of services, processes, network listeners, and the step‑by‑step remediation measures including script‑based scans, permission hardening, and upgrading Struts2.
This article outlines a comprehensive PDCA‑based methodology for e‑commerce platforms to proactively prevent issues, quickly detect anomalies, and execute rapid decisions during large‑scale promotions, covering system goal definition, performance evaluation, capacity planning, SLA management, and team/process maturity.
During the Chinese New Year a client’s Oracle‑Tomcat server was overwhelmed by massive UDP traffic, prompting a forensic investigation that uncovered a hidden Trojan, detailed command‑line analysis, iptables hardening, and the root cause of a weak SSH password left after a hardware upgrade.
This article shares YY's security operations journey, detailing real incident response scenarios, the evolution of their security infrastructure from 2012 onward, and the key factors considered when building a robust security ops system, including DDoS protection, WAF, vulnerability scanning, intrusion detection, and data‑driven automation.
A detailed post‑mortem recounts how a forgotten domain renewal caused a DNS outage, the frantic troubleshooting steps across teams, temporary work‑arounds like switching to Google DNS, and the lessons learned for future incident management.
This article recounts the GitLab production database deletion incident, analyzes why backup mechanisms failed, shares technical and cultural lessons on operational practices, and offers concrete recommendations for building resilient, high‑availability systems to prevent data loss.
The GitLab production database was mistakenly deleted during a manual fix, exposing gaps in backup strategies, PostgreSQL configuration, and operational practices, and prompting a detailed post‑mortem that highlights the need for automated recovery, proper tooling, and transparent incident handling.
When a production server is compromised, abrupt actions like pulling the plug can disrupt services, so this guide outlines an eight‑stage, evidence‑driven response process—including verification, on‑site preservation, containment, impact assessment, online analysis, backup, deep forensics, and reporting—plus real‑world case studies and concrete command examples.
This article recounts a real‑world incident on an Ubuntu 12.04 server where massive outbound traffic was traced to a hidden trojan, detailing step‑by‑step investigation, identification of malicious processes, removal techniques, and preventive hardening measures.
The discussion outlines essential service‑monitoring techniques—including health checks, JVM metrics, traffic and payment ring‑ratio analysis, client‑side exception tracking, third‑party CDN monitoring, alert thresholds, instrumentation via AOP or SDKs, and tooling such as Datadog, Zabbix, and the Elastic stack—to reliably detect and respond to incidents in e‑commerce environments.
This guide teaches operations engineers a four‑step “Wang‑Wen‑Wen‑Qie” methodology—observing system health, listening via alerts, questioning changes, and pulse‑checking with profiling tools—to rapidly diagnose and resolve high‑impact incidents while maintaining clear communication and post‑mortem learning.
In a two‑day incident, Tencent Cloud's CBS team diagnosed cell failures, implemented directed reads and a dual‑cell merge strategy, and restored three‑copy data integrity, while uncovering monitoring gaps and tool limitations that inform future storage operations.
Learn how to systematically examine a Linux host for compromise by checking logs, user accounts, processes, file integrity, network activity, scheduled tasks, services, and rootkits, using built‑in commands and RPM verification to uncover hidden threats.
This article, excerpted from the upcoming Chinese edition of “SRE: Google Site Reliability Engineering”, examines how Google’s SRE guiding philosophies—disaster planning, post‑mortem culture, automation, and data‑driven decision‑making—are adopted, adapted, or contrasted in sectors such as manufacturing, aerospace, nuclear, telecommunications, healthcare, and finance, highlighting key similarities, differences, and lessons for Google and the broader tech industry.
The article explains how weak passwords, misconfigured services like Redis, careless port changes, and leaked data enable attackers to compromise servers and internal networks, illustrating each risk with real‑world case studies and offering practical mitigation advice for robust ops security.
This article shares practical DBA advice—pre‑operation preparation, thorough fault analysis, effective communication, mandatory backups, and post‑incident reviews—to help database administrators maintain stability and avoid costly mistakes during online operations.
This article walks through a real-world Linux server compromise, detailing the attack symptoms, forensic analysis steps, rootkit discovery, exploitation of an Awstats script vulnerability, and practical remediation measures to restore and harden the affected system.
This article outlines why proactive information‑security preparedness, cross‑department response teams, and clear incident‑response checklists are essential for minimizing damage and maintaining trust when a data breach occurs.
This article outlines practical communication techniques, environment choices, active listening, and emotional control for operations teams, then details how to define standards, build robust processes, and ensure reliable business continuity through clear responsibilities, monitoring, automation, and continuous improvement.
A sudden shutdown of an Alibaba Cloud server revealed it had been hijacked as a bot, prompting a step‑by‑step remediation that highlights the importance of basic security hardening such as securing Redis, changing default SSH settings, and enabling cloud monitoring alerts.
The article outlines five subtle indicators of a web application breach—abnormal behavior, irregular logs, unexpected processes or users, file modifications, and warning messages—while offering practical monitoring and remediation steps to help security teams detect and mitigate attacks early.
This article explores the characteristics, causes, and practical strategies for managing difficult or “thorny” team members in operations, offering case studies and step‑by‑step recommendations to mitigate risks while maintaining team performance.
This guide shares ten practical fault‑management techniques—ranging from proactive attitude and prioritizing incidents to continuous follow‑up and team collaboration—to help operations teams reduce damage, maintain service reliability, and keep users engaged during outages.
This guide provides a comprehensive 11‑step Linux security inspection checklist, covering account verification, log analysis, process and file checks, package integrity, network monitoring, scheduled tasks, backdoor detection, kernel modules, services, and rootkit scanning to help identify system compromises.
