How to Respond When Your Server Is Compromised: Essential Incident Response and Forensics for Ops

Invasion Signal Identification

Common signs that a server may have been breached include abnormal CPU or memory usage, sudden disk consumption, unexpected login entries, unknown user accounts or privilege changes, and suspicious network connections.

CPU or memory usage spikes without explanation

Large, unexplained disk space consumption

Irregular login records in system logs

New unknown user accounts or permission changes

Network Anomalies

# Check abnormal network connections
netstat -antp | grep ESTABLISHED
ss -tulpn | grep :22
# Monitor network traffic
iftop -i eth0
nethogs eth0

Process Anomalies

# List top CPU‑consuming processes
ps aux --sort=-%cpu | head -20
# Look for mining or crypto processes
ps aux | grep -E "(bitcoin|mining|crypto)" | grep -v grep
# Check process start times
ps -eo pid,ppid,cmd,etime | sort -k4

Quick Detection Script

#!/bin/bash

echo "=== System Intrusion Detection Report ==="
echo "Detection Time: $(date)"

# Recent logins
echo -e "
[Recent Login Records]"
last | head -10

# Top CPU‑consuming processes
echo -e "
[Top 10 CPU Processes]"
ps aux --sort=-%cpu | head -11

# External connections statistics
echo -e "
[External Connection Stats]"
netstat -an | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

# System file checks
echo -e "
[System File Check]"
find /bin /usr/bin /sbin /usr/sbin -type f -newer /boot/grub/grub.conf 2>/dev/null | head -10

24‑Hour Incident Response Process

Phase 1: Immediate Isolation (0‑30 min)

Disconnect the network while keeping the host powered on to preserve volatile data.

# Drop all inbound and outbound traffic
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP
# Or physically disconnect the NIC
ifconfig eth0 down

Create a dedicated incident directory and record the current system state.

mkdir -p /tmp/incident_$(date +%Y%m%d_%H%M%S)
cd /tmp/incident_$(date +%Y%m%d_%H%M%S)

date > timeline.txt
who >> timeline.txt
w   >> timeline.txt

Phase 2: Information Gathering (30 min‑2 h)

Collect memory images, process lists, network connections, and system configuration.

# Memory dump (if space permits)
dd if=/dev/mem of=memory_dump.img bs=1M
# Or use volatility
yum install volatility -y
volatility -f memory_dump.img imageinfo

# Process evidence
ps auxwww > processes.txt
pstree -p > process_tree.txt
lsof > open_files.txt

# Network connection details
netstat -antp > network_connections.txt
ss -tuln > socket_stats.txt

Phase 3: Log Analysis (2‑6 h)

Gather critical logs from the system, web server, and applications, then search for suspicious entries.

# Copy essential logs
cp /var/log/messages logs/
cp /var/log/secure logs/
cp /var/log/auth.log logs/ 2>/dev/null
cp /var/log/nginx/access.log logs/ 2>/dev/null
cp /var/log/apache2/access.log logs/ 2>/dev/null
find /var/log -name "*.log" -mtime -7 -exec cp {} logs/ \;

# Analyze failed logins
grep -i "failed\|failure\|invalid" /var/log/secure | tail -50
grep "Accepted password" /var/log/secure | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Detect web attacks
grep -E "(union|select|drop|insert|update|delete)" /var/log/nginx/access.log
grep -E "(\../|etc/passwd|/bin/sh)" /var/log/nginx/access.log

# Find suspicious PHP files
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*\$_\|passthru.*\$_" {} \;

Digital Forensics: Tracing the Attack

File‑System Forensics

Identify recently modified or created files and check for SUID/SGID binaries.

# Files modified in the last week
find / -type f -mtime -7 -ls | sort -k8,9
# Files created in the last day (excluding system dirs)
find / -type f -ctime -1 2>/dev/null | grep -vE "/proc|/sys|/dev"
# SUID/SGID checks
find / -perm -4000 -type f -exec ls -la {} \; 2>/dev/null
find / -perm -2000 -type f -exec ls -la {} \; 2>/dev/null

Webshell Detection

# Detect PHP webshell patterns
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*\$_\|passthru.*\$_" {} \;
# One‑liner backdoor detection
grep -r "eval(\$_POST" /var/www/
grep -r "assert(\$_POST" /var/www/
grep -r "preg_replace.*\/e" /var/www/
# Find world‑writable files
find /var/www -type f -perm -o+w -exec ls -la {} \;

Network Forensics

Capture traffic with tcpdump and analyze DNS or HTTP flows.

# Capture all traffic
tcpdump -i any -w traffic_$(date +%H%M%S).pcap &
# DNS queries
tcpdump -i any port 53 -w dns_traffic.pcap
# HTTP traffic
tcpdump -i any port 80 -A -s 0 | grep -E "(GET|POST|User-Agent)"

Historical Connection Analysis

# Bash history for all users
for user in $(cut -d: -f1 /etc/passwd); do
  echo "=== $user 的命令历史 ==="
  cat /home/$user/.bash_history 2>/dev/null | tail -50
done

# Check SSH authorized keys
find /home -name "authorized_keys" -exec echo "=== {} ===" \; -exec cat {} \;
find /home -name "id_rsa*" -exec ls -la {} \;

Cleanup and Hardening

Threat Removal

# Kill suspicious processes (e.g., mining)
kill -9 $(ps aux | grep -E "(bitcoin|mining|crypto)" | grep -v grep | awk '{print $2}')
# Remove cron jobs
crontab -r
echo "" > /etc/crontab
rm -rf /var/spool/cron/*

File Cleanup

# Delete detected webshells
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode" {} \; | xargs rm -f
# Clean temporary files
find /tmp -type f -mtime -7 -exec rm -f {} \;
find /var/tmp -type f -mtime -7 -exec rm -f {} \;
# Reset permissions
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

System Hardening

Enforce strong passwords, disable unnecessary accounts, and tighten SSH configuration.

# Change passwords
passwd root
passwd $(whoami)
# Disable unused users
usermod -s /sbin/nologin apache
usermod -s /sbin/nologin nginx
# SSH hardening
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Configure a restrictive firewall to allow only required services.

# Flush existing rules
iptables -F
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback and established connections
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Open necessary ports
iptables -A INPUT -p tcp --dport 22 -j ACCEPT   # SSH
iptables -A INPUT -p tcp --dport 80 -j ACCEPT   # HTTP
iptables -A INPUT -p tcp --dport 443 -j ACCEPT  # HTTPS

service iptables save

Prevention: Building a Monitoring System

Real‑Time Monitoring Script

#!/bin/bash
ALERT_EMAIL="[email protected]"
LOG_FILE="/var/log/security_monitor.log"

check_suspicious_processes() {
  MINING_PROCS=$(ps aux | grep -E "(bitcoin|mining|crypto|xmrig)" | grep -v grep)
  if [ -n "$MINING_PROCS" ]; then
    echo "$(date): Detected mining processes: $MINING_PROCS" >> $LOG_FILE
    echo "检测到挖矿进程" | mail -s "安全警报" $ALERT_EMAIL
  fi

  HIGH_CPU=$(ps aux --sort=-%cpu | awk 'NR>1 && $3>80 {print $0}')
  if [ -n "$HIGH_CPU" ]; then
    echo "$(date): High CPU processes: $HIGH_CPU" >> $LOG_FILE
  fi
}

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/secure | grep "$(date +%b\ %d)" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Failed login attempts today: $FAILED_COUNT" >> $LOG_FILE
    echo "检测到暴力破解攻击" | mail -s "登录安全警报" $ALERT_EMAIL
  fi
}

while true; do
  check_suspicious_processes
  check_failed_logins
  sleep 300
done

File Integrity Monitoring

# Install AIDE
yum install aide -y
# Initialize database
aide --init
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
# Daily check via cron
cat > /etc/cron.daily/aide-check <<'EOF'
#!/bin/bash
aide --check > /tmp/aide-report.txt 2>&1
if [ $? -ne 0 ]; then
  mail -s "文件完整性检查异常" [email protected] < /tmp/aide-report.txt
fi
EOF
chmod +x /etc/cron.daily/aide-check

Key Takeaways

Detect quickly – Deploy monitoring and alerts to spot anomalies early.

Isolate precisely – Cut network access immediately to stop lateral movement.

Collect evidence thoroughly – Preserve logs, memory images, and system snapshots for analysis and legal compliance.

Regular patching, strong password policies, least‑privilege access, comprehensive log collection, and rehearsed incident‑response playbooks are essential to keep servers secure.