How to Detect and Respond to Server Intrusions: A Complete 24‑Hour Incident Response Guide

Server Compromised? Essential Incident Response and Forensics for Operations

Warning: Abnormal processes, suspicious network connections, or unknown files may indicate a server breach!

For operations engineers, the biggest fear is not a crash but a hacker intrusion. Mishandling can cause data leaks and legal liability. This article shares a complete response workflow and practical forensics methods to protect enterprise assets.

🚨 Intrusion Signal Detection: Early Discovery, Early Action

Common Intrusion Indicators

System‑level anomalies:

CPU or memory usage spikes without explanation

Sudden massive disk space consumption

Unexpected login records in system logs

New unknown user accounts or privilege changes

Network‑level anomalies:

# Check abnormal network connections
netstat -antp | grep ESTABLISHED
ss -tulpn | grep :22
# Monitor network traffic anomalies
iftop -i eth0
nethogs eth0

Process‑level anomalies:

# View suspicious processes
ps aux --sort=-%cpu | head -20
top -c | grep -E "(bitcoin|mining|crypto)"
# Check process start times
ps -eo pid,ppid,cmd,etime | sort -k4

Quick Detection Script

#!/bin/bash
# Intrusion detection quick script

echo "=== System Intrusion Detection Report ==="
echo "Detection Time: $(date)"

# Recent logins
last | head -10

# High CPU processes
ps aux --sort=-%cpu | head -11

# External connections statistics
netstat -an | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

# System file check (newer than grub config)
find /bin /usr/bin /sbin /usr/sbin -type f -newer /boot/grub/grub.conf 2>/dev/null | head -10

⚡ Golden 24‑Hour Incident Response Process

Phase 1: Immediate Isolation (0‑30 min)

Disconnect network but keep the machine on:

# Preserve memory, isolate network
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP

# Or physically disconnect
ifconfig eth0 down

Preserve the scene:

# Create incident directory
mkdir -p /tmp/incident_$(date +%Y%m%d_%H%M%S)
cd /tmp/incident_$(date +%Y%m%d_%H%M%S)

# Record current state
date > timeline.txt
who >> timeline.txt
w >> timeline.txt

Phase 2: Information Gathering (30 min‑2 h)

Memory forensics:

# Memory image (if enough space)
dd if=/dev/mem of=memory_dump.img bs=1M
# Or use volatility
yum install volatility -y
volatility -f memory_dump.img imageinfo

Process forensics:

# Detailed process info
ps auxwww > processes.txt
pstree -p > process_tree.txt
lsof > open_files.txt

# Network connection details
netstat -antp > network_connections.txt
ss -tuln > socket_stats.txt

System state snapshot:

# System information collection script
#!/bin/bash
INCIDENT_DIR="/tmp/incident_$(date +%Y%m%d_%H%M%S)"
collect_system_info() {
  echo "Collecting system info..."
  uname -a > $INCIDENT_DIR/system_info.txt
  cat /proc/version >> $INCIDENT_DIR/system_info.txt
  uptime >> $INCIDENT_DIR/system_info.txt
  cat /etc/passwd > $INCIDENT_DIR/users.txt
  cat /etc/group > $INCIDENT_DIR/groups.txt
  lastlog > $INCIDENT_DIR/lastlog.txt
  systemctl list-units --type=service > $INCIDENT_DIR/services.txt
  crontab -l > $INCIDENT_DIR/crontab.txt 2>/dev/null
  cat /etc/crontab >> $INCIDENT_DIR/crontab.txt 2>/dev/null
  ifconfig > $INCIDENT_DIR/network_config.txt
  route -n >> $INCIDENT_DIR/network_config.txt
  cat /etc/hosts >> $INCIDENT_DIR/network_config.txt
}

Phase 3: Log Analysis (2‑6 h)

Key log collection:

# System logs
cp /var/log/messages logs/
cp /var/log/secure logs/
cp /var/log/auth.log logs/ 2>/dev/null

# Web server logs
cp /var/log/nginx/access.log logs/ 2>/dev/null
cp /var/log/apache2/access.log logs/ 2>/dev/null

# Application logs (last 7 days)
find /var/log -name "*.log" -mtime -7 -exec cp {} logs/ \;

Log analysis tips:

# Analyze suspicious logins
grep -i "failed\|failure\|invalid" /var/log/secure | tail -50
grep "Accepted password" /var/log/secure | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Analyze web attacks
grep -E "(union|select|drop|insert|update|delete)" /var/log/nginx/access.log
grep -E "(\./|etc/passwd|/bin/sh)" /var/log/nginx/access.log

# Analyze abnormal file access
find / -name "*.php" -mtime -1 2>/dev/null | xargs grep -l "eval\|base64_decode\|shell_exec"

🔍 Digital Forensics: Tracing the Attack

File System Forensics

Timeline analysis:

# Find suspicious files by modification time
find / -type f -mtime -7 -ls 2>/dev/null | sort -k8,9

# Find recently created files (exclude system dirs)
find / -type f -ctime -1 2>/dev/null | grep -v "/proc\|/sys\|/dev"

# SUID/SGID file check
find / -perm -4000 -type f -exec ls -la {} \; 2>/dev/null
find / -perm -2000 -type f -exec ls -la {} \; 2>/dev/null

Webshell detection:

# PHP webshell signature detection
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*$_\|passthru.*$_" {} \;
# One‑line backdoor detection
grep -r "eval($_POST" /var/www/
grep -r "assert($_POST" /var/www/
grep -r "preg_replace.*\/e" /var/www/
# File permission anomalies
find /var/www -type f -perm -o+w -exec ls -la {} \;

Network Forensics

Traffic analysis tools:

# Capture packets with tcpdump
tcpdump -i any -w traffic_$(date +%H%M%S).pcap &
# DNS query analysis
tcpdump -i any port 53 -w dns_traffic.pcap
# HTTP traffic analysis
tcpdump -i any port 80 -A -s 0 | grep -E "(GET|POST|User-Agent)"

Historical connection analysis:

# Analyze bash history for each user
for user in $(cat /etc/passwd | cut -d: -f1); do
  echo "=== $user command history ==="
  cat /home/$user/.bash_history 2>/dev/null | tail -50
done
# Check SSH keys
find /home -name "authorized_keys" -exec echo "=== {} ===" \; -exec cat {} \;
find /home -name "id_rsa*" -exec ls -la {} \;

🛡️ Cleanup and Hardening: Eradicating the Threat

Threat Removal

Process cleanup:

# Terminate suspicious processes
kill -9 $(ps aux | grep -E "(bitcoin|mining|crypto)" | awk '{print $2}')
# Clean scheduled tasks
crontab -r
echo "" > /etc/crontab
rm -rf /var/spool/cron/*

File cleanup:

# Delete webshells
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode" {} \; | xargs rm -f
# Clean temporary files
find /tmp -type f -mtime -7 -exec rm -f {} \;
find /var/tmp -type f -mtime -7 -exec rm -f {} \;
# Reset file permissions
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

System Hardening

Account security:

# Change passwords
passwd root
passwd $(whoami)
# Disable unnecessary users
usermod -s /sbin/nologin apache
usermod -s /sbin/nologin nginx
# SSH hardening
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

Firewall configuration:

# Flush rules
iptables -F
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

service iptables save

📊 Proactive Monitoring Framework

Real‑time Monitoring Script

#!/bin/bash
# Server security monitoring script
ALERT_EMAIL="[email protected]"
LOG_FILE="/var/log/security_monitor.log"

check_suspicious_processes() {
  MINING_PROCS=$(ps aux | grep -E "(bitcoin|mining|crypto|xmrig)" | grep -v grep)
  if [ -n "$MINING_PROCS" ]; then
    echo "$(date): Detected mining processes: $MINING_PROCS" >> $LOG_FILE
    echo "Mining process detected" | mail -s "Security Alert" $ALERT_EMAIL
  fi
  HIGH_CPU=$(ps aux --sort=-%cpu | awk 'NR>1 && $3>80 {print $0}')
  if [ -n "$HIGH_CPU" ]; then
    echo "$(date): High CPU processes: $HIGH_CPU" >> $LOG_FILE
  fi
}

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/secure | grep "$(date +%b\ %d)" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Failed login attempts today: $FAILED_COUNT" >> $LOG_FILE
    echo "Brute‑force attack detected" | mail -s "Login Alert" $ALERT_EMAIL
  fi
}

while true; do
  check_suspicious_processes
  check_failed_logins
  sleep 300
done

File Integrity Monitoring

# Install AIDE for integrity checking
yum install aide -y
# Initialize database
aide --init
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
# Daily check script
cat > /etc/cron.daily/aide-check <<'EOF'
#!/bin/bash
aide --check > /tmp/aide-report.txt 2>&1
if [ $? -ne 0 ]; then
  mail -s "File Integrity Check Alert" [email protected] < /tmp/aide-report.txt
fi
EOF
chmod +x /etc/cron.daily/aide-check

🎯 Summary: Core Principles of Operations Security

Three essentials of rapid response:

Detect quickly – Deploy monitoring and alerts to spot anomalies instantly.

Isolate precisely – Cut network access immediately to prevent lateral spread.

Forensically capture fully – Preserve complete evidence for later analysis.

Daily protection focus:

Regularly apply system patches and software updates.

Enforce least‑privilege and network segmentation.

Maintain comprehensive log collection and analysis.

Develop and rehearse incident‑response playbooks.

Lessons learned: Most breaches stem from management failures, not technology. Over 90% can be prevented with basic security measures: timely patching, strong passwords, strict access control, and continuous log monitoring.