How to Detect and Respond to Server Intrusions: A Complete 24‑Hour Playbook

Invasion Signal Identification: Early Detection

Common Invasion Signs

CPU or memory usage spikes without explanation

Sudden massive disk consumption

Abnormal login records in system logs

New unknown user accounts or privilege changes

Network Layer Anomalies

# Check abnormal network connections
netstat -antp | grep ESTABLISHED
ss -tulpn | grep :22  # Check SSH connections

# Monitor network traffic anomalies
iftop -i eth0
nethogs eth0

Process Layer Anomalies

# View suspicious processes
ps aux --sort=-%cpu | head -20
top -c | grep -E "(bitcoin|mining|crypto)"

# Check process start times
ps -eo pid,ppid,cmd,etime | sort -k4

Quick Detection Script

#!/bin/bash
# Intrusion detection quick script

echo "=== System Intrusion Detection Report ==="
echo "Detection Time: $(date)"

# Recent logins
echo -e "
[Recent Login Records]"
last | head -10

# Top CPU consuming processes
echo -e "
[Top 10 CPU Processes]"
ps aux --sort=-%cpu | head -11

# External connection statistics
echo -e "
[External Connections]"
netstat -an | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

# System file check
echo -e "
[System Files]"
find /bin /usr/bin /sbin /usr/sbin -type f -newer /boot/grub/grub.conf 2>/dev/null | head -10

⚡ 24‑Hour Emergency Response Process

Phase 1: Immediate Isolation (0‑30 min)

Disconnect network but keep the machine running:

# Preserve memory data, isolate network
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP

# Or physically disconnect
ifconfig eth0 down

Preserve the scene:

# Create incident directory
mkdir -p /tmp/incident_$(date +%Y%m%d_%H%M%S)
cd /tmp/incident_$(date +%Y%m%d_%H%M%S)

# Record current state
date > timeline.txt
who >> timeline.txt
w >> timeline.txt

Phase 2: Information Gathering (30 min‑2 h)

Memory acquisition (if space permits):

# Memory dump
dd if=/dev/mem of=memory_dump.img bs=1M
# Or use volatility
yum install volatility -y
volatility -f memory_dump.img imageinfo

Process forensics:

# Detailed process info
ps auxwww > processes.txt
pstree -p > process_tree.txt
lsof > open_files.txt

# Network connection details
netstat -antp > network_connections.txt
ss -tuln > socket_stats.txt

System snapshot:

#!/bin/bash
INCIDENT_DIR="/tmp/incident_$(date +%Y%m%d_%H%M%S)"

collect_system_info() {
  echo "Collecting system information..."
  uname -a > $INCIDENT_DIR/system_info.txt
  cat /proc/version >> $INCIDENT_DIR/system_info.txt
  uptime >> $INCIDENT_DIR/system_info.txt
  cat /etc/passwd > $INCIDENT_DIR/users.txt
  cat /etc/group > $INCIDENT_DIR/groups.txt
  lastlog > $INCIDENT_DIR/lastlog.txt
  systemctl list-units --type=service > $INCIDENT_DIR/services.txt
  crontab -l > $INCIDENT_DIR/crontab.txt 2>/dev/null
  cat /etc/crontab >> $INCIDENT_DIR/crontab.txt 2>/dev/null
  ifconfig > $INCIDENT_DIR/network_config.txt
  route -n >> $INCIDENT_DIR/network_config.txt
  cat /etc/hosts >> $INCIDENT_DIR/network_config.txt
}

collect_system_info

Phase 3: Log Analysis (2‑6 h)

Key log collection:

# System logs
cp /var/log/messages logs/
cp /var/log/secure logs/
cp /var/log/auth.log logs/ 2>/dev/null

# Web server logs
cp /var/log/nginx/access.log logs/ 2>/dev/null
cp /var/log/apache2/access.log logs/ 2>/dev/null

# Application logs (last 7 days)
find /var/log -name "*.log" -mtime -7 -exec cp {} logs/ \;

Log analysis techniques:

# Analyze suspicious logins
grep -i "failed\|failure\|invalid" /var/log/secure | tail -50
grep "Accepted password" /var/log/secure | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Analyze web attacks
grep -E "(union|select|drop|insert|update|delete)" /var/log/nginx/access.log
grep -E "(\./\.|etc/passwd|/bin/sh)" /var/log/nginx/access.log

# Detect malicious PHP files
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*\$_\|passthru.*\$_" {} \;

# Find recent PHP files with suspicious functions
find / -name "*.php" -mtime -1 2>/dev/null | xargs grep -l "eval\|base64_decode\|shell_exec"

# Detect suspicious file permissions
find /var/www -type f -perm -o+w -exec ls -la {} \;

🔍 Digital Forensics: Tracing the Attack

File System Forensics

Timeline analysis:

# Find recently modified files (last 7 days)
find / -type f -mtime -7 -ls | sort -k8,9

# Find files created within the last day, excluding system dirs
find / -type f -ctime -1 2>/dev/null | grep -v "/proc\|/sys\|/dev"

# Check SUID/SGID binaries
find / -perm -4000 -type f -exec ls -la {} \; 2>/dev/null
find / -perm -2000 -type f -exec ls -la {} \; 2>/dev/null

Webshell Detection

# Detect PHP webshell signatures
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode\|system.*\$_\|passthru.*\$_" {} \;

# Detect one‑liner backdoors
grep -r "eval(\$_POST" /var/www/
grep -r "assert(\$_POST" /var/www/
grep -r "preg_replace.*\/e" /var/www/

# Find files with world‑writable permissions
find /var/www -type f -perm -o+w -exec ls -la {} \;

Network Forensics

Traffic analysis tools:

# Capture packets with tcpdump
tcpdump -i any -w traffic_$(date +%H%M%S).pcap &

# Analyze DNS queries
tcpdump -i any port 53 -w dns_traffic.pcap

# HTTP traffic analysis
tcpdump -i any port 80 -A -s 0 | grep -E "(GET|POST|User-Agent)"

# Historical connection analysis (bash history)
for user in $(cut -d: -f1 /etc/passwd); do
  echo "=== $user command history ==="
  cat /home/$user/.bash_history 2>/dev/null | tail -50
  done

# Check SSH keys
find /home -name "authorized_keys" -exec echo "=== {} ===" \; -exec cat {} \;
find /home -name "id_rsa*" -exec ls -la {} \;

🛡️ Cleanup and Hardening: Eradicating the Threat

Threat Removal

Process cleanup:

# Terminate suspicious processes
kill -9 $(ps aux | grep -E "(bitcoin|mining|crypto)" | awk '{print $2}')

# Remove malicious cron jobs
crontab -r
> /etc/crontab
rm -rf /var/spool/cron/*

File cleanup:

# Delete webshells
find /var/www -name "*.php" -exec grep -l "eval.*base64_decode" {} \; | xargs rm -f

# Clean temporary files
find /tmp -type f -mtime -7 -exec rm -f {} \;
find /var/tmp -type f -mtime -7 -exec rm -f {} \;

# Reset file permissions
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

System Hardening

Account security:

# Change all passwords
passwd root
passwd $(whoami)

# Disable unnecessary users
usermod -s /sbin/nologin apache
usermod -s /sbin/nologin nginx

# SSH security configuration
sed -i 's/#PermitRootLogin\s*yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication\s*yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

Firewall configuration (iptables):

iptables -F
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

service iptables save

📊 Proactive Monitoring Framework

Real‑time Monitoring Script

#!/bin/bash
# Server security monitoring script

ALERT_EMAIL="[email protected]"
LOG_FILE="/var/log/security_monitor.log"

check_suspicious_processes() {
  MINING_PROCS=$(ps aux | grep -E "(bitcoin|mining|crypto|xmrig)" | grep -v grep)
  if [ ! -z "$MINING_PROCS" ]; then
    echo "$(date): Detected mining processes: $MINING_PROCS" >> $LOG_FILE
    echo "Detected mining processes" | mail -s "Security Alert" $ALERT_EMAIL
  fi

  HIGH_CPU=$(ps aux --sort=-%cpu | awk 'NR>1 && $3>80 {print $0}')
  if [ ! -z "$HIGH_CPU" ]; then
    echo "$(date): High CPU processes: $HIGH_CPU" >> $LOG_FILE
  fi
}

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/secure | grep "$(date +%b\ %d)" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Failed login attempts today: $FAILED_COUNT" >> $LOG_FILE
    echo "Brute‑force login detected" | mail -s "Login Security Alert" $ALERT_EMAIL
  fi
}

while true; do
  check_suspicious_processes
  check_failed_logins
  sleep 300
done

File Integrity Monitoring

# Install AIDE
yum install aide -y

# Initialize database
aide --init
mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# Daily check via cron
cat > /etc/cron.daily/aide-check <<'EOF'
#!/bin/bash
aide --check > /tmp/aide-report.txt 2>&1
if [ $? -ne 0 ]; then
  mail -s "File integrity check alert" [email protected] < /tmp/aide-report.txt
fi
EOF
chmod +x /etc/cron.daily/aide-check

🎯 Summary: Core Principles for Operations Security

Detect quickly – Deploy monitoring and alerts to spot anomalies as soon as they appear.

Isolate precisely – Cut network access immediately to stop lateral movement.

Collect evidence thoroughly – Preserve logs, memory dumps, and file snapshots for later analysis.

Daily protection focus:

Regularly apply system patches and update software.

Enforce least‑privilege principles and network segmentation.

Maintain comprehensive log collection and automated analysis.

Develop and rehearse an incident‑response playbook.

Lesson learned: Most breaches stem from basic management failures; 90% can be prevented with timely patches, strong passwords, strict access control, and continuous monitoring.