Essential Incident Response & Forensics Guide for Server Intrusions

Server Intrusion: Emergency Response Process and Forensic Methods

🚨 Invasion Signal Identification: Early Detection

Common Invasion Indicators

System‑level anomalies:

CPU or memory usage spikes without explanation

Sudden massive disk consumption

Abnormal login records in system logs

New unknown user accounts or privilege changes

Network‑level anomalies:

# Check abnormal network connections
netstat -antp | grep ESTABLISHED
ss -tulpn | grep :22  # Check SSH connections

# Monitor network traffic anomalies
iftop -i eth0
nethogs eth0

Process‑level anomalies:

# View suspicious processes
ps aux --sort=-%cpu | head -20
top -c | grep -E "(bitcoin|mining|crypto)"

# Check process start times
ps -eo pid,ppid,cmd,etime | sort -k4

Quick Detection Script

#!/bin/bash
# Intrusion detection quick script

echo "=== System Intrusion Detection Report ==="
echo "Detection Time: $(date)"

# Recent logins
echo -e "
[Recent Login Records]"
last | head -10

# High‑CPU processes
echo -e "
[Top 10 CPU Processes]"
ps aux --sort=-%cpu | head -11

# External connections
echo -e "
[External Connection Statistics]"
netstat -an | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

# System file check
echo -e "
[System File Check]"
find /bin /usr/bin /sbin /usr/sbin -type f -newer /boot/grub/grub.conf 2>/dev/null | head -10

⚡ Golden 24‑Hour: Emergency Response Workflow

Phase 1 – Immediate Isolation (0‑30 min)

Disconnect network but keep the machine powered on:

# Preserve memory, isolate network
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP
# Or physically disconnect
ifconfig eth0 down

Preserve the scene:

# Create incident directory
mkdir -p /tmp/incident_$(date +%Y%m%d_%H%M%S)
cd /tmp/incident_$(date +%Y%m%d_%H%M%S)

date > timeline.txt
who >> timeline.txt
w >> timeline.txt

Phase 2 – Information Gathering (30 min‑2 h)

Memory acquisition (if space permits):

# Memory dump
dd if=/dev/mem of=memory_dump.img bs=1M
# Or use volatility
yum install volatility -y
volatility -f memory_dump.img imageinfo

Process evidence:

ps auxwww > processes.txt
pstree -p > process_tree.txt
lsof > open_files.txt
netstat -antp > network_connections.txt
ss -tuln > socket_stats.txt

System snapshot:

# System info collection script
#!/bin/bash
INCIDENT_DIR="/tmp/incident_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$INCIDENT_DIR"

echo "Collecting system info..."
uname -a > "$INCIDENT_DIR/system_info.txt"
cat /proc/version >> "$INCIDENT_DIR/system_info.txt"
uptime >> "$INCIDENT_DIR/system_info.txt"
cat /etc/passwd > "$INCIDENT_DIR/users.txt"
cat /etc/group > "$INCIDENT_DIR/groups.txt"
lastlog > "$INCIDENT_DIR/lastlog.txt"
systemctl list-units --type=service > "$INCIDENT_DIR/services.txt"
crontab -l > "$INCIDENT_DIR/crontab.txt" 2>/dev/null
ifconfig > "$INCIDENT_DIR/network_config.txt"
route -n >> "$INCIDENT_DIR/network_config.txt"
cat /etc/hosts >> "$INCIDENT_DIR/network_config.txt"

Phase 3 – Log Analysis (2‑6 h)

Key log collection:

# System logs
cp /var/log/messages logs/
cp /var/log/secure logs/
cp /var/log/auth.log logs/ 2>/dev/null
# Web server logs
cp /var/log/nginx/access.log logs/ 2>/dev/null
cp /var/log/apache2/access.log logs/ 2>/dev/null
# Application logs (last 7 days)
find /var/log -name "*.log" -mtime -7 -exec cp {} logs/ \;

Log analysis tips:

# Suspicious login analysis
grep -i "failed\|failure\|invalid" /var/log/secure | tail -50
grep "Accepted password" /var/log/secure | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Web attack patterns
grep -E "(union|select|drop|insert|update|delete)" /var/log/nginx/access.log
grep -E "(\../|etc/passwd|/bin/sh)" /var/log/nginx/access.log

# Suspicious file access
find / -name "*.php" -mtime -1 -exec grep -l "eval\|base64_decode\|shell_exec" {} \;

🔍 Digital Forensics: Tracing the Attack Path

File System Forensics

Timeline analysis:

# Find recently modified files (last 7 days)
find / -type f -mtime -7 -ls | sort -k8,9

# Find files created within the last day (exclude system dirs)
find / -type f -ctime -1 2>/dev/null | grep -vE "/proc|/sys|/dev"

# SUID/SGID checks
find / -perm -4000 -type f -exec ls -la {} \; 2>/dev/null
find / -perm -2000 -type f -exec ls -la {} \; 2>/dev/null

Webshell detection:

# Detect PHP webshell signatures
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*\$_\|passthru.*\$_" {} \;
# One‑liner backdoors
grep -r "eval(\$_POST" /var/www/
grep -r "assert(\$_POST" /var/www/
grep -r "preg_replace.*\/e" /var/www/
# Files with world‑writable permissions
find /var/www -type f -perm -o+w -exec ls -la {} \;

Network Forensics

Traffic analysis tools:

# Capture all traffic
tcpdump -i any -w traffic_$(date +%H%M%S).pcap &
# DNS queries
tcpdump -i any port 53 -w dns_traffic.pcap
# HTTP traffic
tcpdump -i any port 80 -A -s 0 | grep -E "(GET|POST|User-Agent)"

Historical connection analysis:

# Bash history for each user
for user in $(cut -d: -f1 /etc/passwd); do
  echo "=== $user command history ==="
  cat /home/$user/.bash_history 2>/dev/null | tail -50
done
# SSH keys
find /home -name "authorized_keys" -exec echo "=== {} ===" \; -exec cat {} \;
find /home -name "id_rsa*" -exec ls -la {} \;

🛡️ Cleanup and Hardening: Eradicating the Threat

Threat Removal

Process cleanup:

# Kill suspicious mining processes
kill -9 $(ps aux | grep -E "(bitcoin|mining|crypto)" | awk '{print $2}')
# Remove malicious cron jobs
crontab -r
echo "" > /etc/crontab
rm -rf /var/spool/cron/*

File cleanup:

# Delete webshells
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode" {} \; | xargs rm -f
# Clean temporary files
find /tmp -type f -mtime -7 -exec rm -f {} \;
find /var/tmp -type f -mtime -7 -exec rm -f {} \;
# Reset permissions
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

System Hardening

Account security:

# Change all passwords
passwd root
passwd $(whoami)
# Disable unnecessary users
usermod -s /sbin/nologin apache
usermod -s /sbin/nologin nginx
# SSH hardening
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

Firewall configuration:

iptables -F
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
service iptables save

📊 Prevention First: Building a Monitoring System

Real‑time Monitoring Script

#!/bin/bash
# Server security monitoring script
ALERT_EMAIL="[email protected]"
LOG_FILE="/var/log/security_monitor.log"

check_suspicious_processes() {
  MINING_PROCS=$(ps aux | grep -E "(bitcoin|mining|crypto|xmrig)" | grep -v grep)
  if [ ! -z "$MINING_PROCS" ]; then
    echo "$(date): Detected mining processes: $MINING_PROCS" >> $LOG_FILE
    echo "Detected mining processes" | mail -s "Security Alert" $ALERT_EMAIL
  fi

  HIGH_CPU=$(ps aux --sort=-%cpu | awk 'NR>1 && $3>80 {print $0}')
  if [ ! -z "$HIGH_CPU" ]; then
    echo "$(date): High CPU processes: $HIGH_CPU" >> $LOG_FILE
  fi
}

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/secure | grep "$(date +%b\ %d)" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Failed login attempts today: $FAILED_COUNT" >> $LOG_FILE
    echo "Brute‑force attack detected" | mail -s "Login Security Alert" $ALERT_EMAIL
  fi
}

while true; do
  check_suspicious_processes
  check_failed_logins
  sleep 300
done

File Integrity Monitoring

# Install AIDE
yum install aide -y
# Initialize database
aide --init
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
# Daily check via cron
cat > /etc/cron.daily/aide-check <<'EOF'
#!/bin/bash
aide --check > /tmp/aide-report.txt 2>&1
if [ $? -ne 0 ]; then
  mail -s "File integrity check alert" [email protected] < /tmp/aide-report.txt
fi
EOF
chmod +x /etc/cron.daily/aide-check

🎯 Summary: Core Principles of Operations Security

Rapid response three keys:

Detect quickly – Deploy monitoring and alerts to spot anomalies instantly.

Isolate precisely – Cut network access immediately to stop lateral movement.

Collect evidence thoroughly – Preserve logs, memory, and artifacts for analysis.

Daily protection focus:

Regularly patch systems and software.

Apply least‑privilege principle and network segmentation.

Maintain comprehensive log collection and analysis.

Establish and rehearse an incident‑response plan.

Lesson learned: Most breaches stem from management gaps; 90% are preventable with basic security hygiene such as timely patches, strong passwords, strict access control, and continuous monitoring.