Master Linux Server Intrusion Detection & Rapid Incident Response: A Complete Hands‑On Guide

Introduction

As a sysadmin, you may receive late‑night alerts about abnormal CPU usage, traffic spikes, or other signs of a possible intrusion. This guide provides a complete workflow for detecting and responding to Linux server compromises, helping you stay organized and act quickly.

1. Intrusion Detection – Early Discovery

1.1 System Anomaly Metrics

Key monitoring items include system load, CPU and memory usage, network connections, and suspicious processes.

# System load detection
uptime                     # view load average
top -bn1 | head -20        # CPU and memory usage

# Network connection checks
netstat -tuln | grep LISTEN   # listening ports
ss -tulnp | grep :22          # SSH connections
netstat -an | grep ESTABLISHED | wc -l   # active connections

# Process anomaly detection
ps aux --sort=-%cpu | head -10   # top CPU consumers
ps aux --sort=-%mem | head -10   # top memory consumers

1.2 Practical Monitoring Scripts

#!/bin/bash
# security_check.sh – quick security audit

echo "=== System Security Report $(date) ==="

# Check suspicious processes
echo "== Suspicious Process Check =="
ps aux | awk '$3 > 80 {print "High‑CPU process: " $0}'
ps aux | awk '$4 > 80 {print "High‑memory process: " $0}'

# Check abnormal network connections
echo "== Network Connection Check =="
netstat -an | grep ":80 " | wc -l | awk '{print "HTTP connections: " $1}'
netstat -an | grep ":22 " | wc -l | awk '{print "SSH connections: " $1}'

# Check failed login attempts
echo "== Login Security Check =="
last -n 10 | grep -v "reboot\|shutdown"
lastb | head -5

1.3 Log Analysis & Anomaly Identification

Core log files to monitor include /var/log/auth.log for authentication events and /var/log/syslog for system warnings.

# Authentication log analysis
tail -f /var/log/auth.log | grep "Failed password"
grep "Accepted password" /var/log/auth.log | tail -10

# System log analysis
tail -f /var/log/syslog | grep -E "(error|warning|critical)"

# Nginx access log (4xx/5xx errors)
tail -f /var/log/nginx/access.log | awk '$9 ~ /^4|^5/ {print $0}'

1.4 One‑Click Log Analyzer

#!/bin/bash
# log_analyzer.sh – automated log anomaly detection

echo "=== Log Security Analysis $(date) ==="

# SSH brute‑force detection
echo "== SSH Brute‑Force Attempts =="
grep "Failed password" /var/log/auth.log | tail -20 | awk '{print $11, $13}' | sort | uniq -c | sort -nr

# Sudo usage review
echo "== Sudo Usage Records =="
grep "sudo:" /var/log/auth.log | tail -10

# New user accounts
echo "== User Account Changes =="
grep "useradd\|userdel" /var/log/auth.log | tail -10

2. Intrusion Confirmation – Identify Attack Type

2.1 Webshell Detection

# Find suspicious PHP files
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode\|shell_exec" {} \;
# List recently modified web files (last 7 days)
find /var/www -type f -mtime -7 -exec ls {} \; | grep -E "\.(php|jsp|asp)$"

2.2 Reverse Shell Detection

# Check for suspicious listening ports
netstat -antp | grep -E ":4444|:5555|:7777|:8888|:9999"
# Inspect bash process network connections
lsof -i -a -p $(pgrep bash)

2.3 Mining Malware Detection

# High‑CPU processes (possible miners)
top -bn1 | awk 'NR>7 && $9>50 {print "Suspicious process:", $12, "CPU:", $9"%"}'
# Known miner process names
ps aux | grep -E "(xmrig|minergate|cpuminer|cgminer)" | grep -v grep

3. Emergency Response – Rapid Damage Control

3.1 Phase 1: Immediate Response (0‑15 min)

1. Isolate the affected system

# Disconnect network (use with caution)
ifconfig eth0 down
# Or block suspicious IPs
iptables -A INPUT -s [suspicious_IP] -j DROP
iptables -A OUTPUT -d [suspicious_IP] -j DROP

2. Terminate malicious processes

# Kill by PID
kill -9 [malicious_PID]
# Kill by name
pkill -f [malicious_name]
# End suspicious user sessions
pkill -u [suspicious_user]

3. Protect critical data

# Backup important logs
cp /var/log/auth.log /tmp/auth.log.backup.$(date +%Y%m%d_%H%M%S)
cp /var/log/syslog /tmp/syslog.backup.$(date +%Y%m%d_%H%M%S)
# Archive configuration files
tar -czf /tmp/config_backup_$(date +%Y%m%d_%H%M%S).tar.gz /etc/

3.2 Phase 2: In‑Depth Investigation (15‑60 min)

1. Attack path analysis

#!/bin/bash
# incident_analysis.sh – trace intrusion timeline

echo "=== Intrusion Path Analysis $(date) ==="

# Attack timeline from auth.log
echo "== Attack Timeline =="
grep -E "(Failed|Accepted)" /var/log/auth.log | tail -50

# Recent file modifications (last 1 day)
find /var/www /tmp /etc -type f -mtime -1 -exec ls -la {} \; | sort -k6,7

# Process creation timeline
ps -eo pid,ppid,cmd,etime,user --sort=etime

2. Vulnerability analysis

# Check for pending OS patches
apt list --upgradable   # Debian/Ubuntu
yum check-update        # CentOS/RHEL

# Verify service versions
nginx -v
apache2 -v
mysql --version
php -v

3.3 Phase 3: System Hardening (1‑4 h)

1. Remove malicious files

# Delete identified webshells
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode" {} \; | xargs rm -f
# Clean recent executable files in /tmp
find /tmp -type f -mtime -1 -exec file {} \; | grep -E "(script|executable)"

2. Account security hardening

# Lock suspicious accounts
usermod -L [suspicious_user]
# Enforce password policy
chage -M 90 -m 1 -W 7 [username]
# Review SSH authorized keys
cat ~/.ssh/authorized_keys

3. Firewall rule optimization

# Basic iptables configuration
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -j DROP
# Save rules
iptables-save > /etc/iptables/rules.v4

4. Preventive Hardening – Building a Secure Perimeter

4.1 Monitoring & Alerting Stack (Prometheus + Grafana)

# prometheus.yml (excerpt)
global:
  scrape_interval: 15s
scrape_configs:
- job_name: 'node-exporter'
  static_configs:
  - targets: ['localhost:9100']
rule_files:
- "security_rules.yml"

# security_rules.yml (example alert)
groups:
- name: security_alerts
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"

4.2 Automated Detection Scripts

#!/bin/bash
# security_monitor.sh – periodic security checks
LOG_FILE="/var/log/security_monitor.log"

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/auth.log | grep "$(date '+%b %d')" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Warning – $FAILED_COUNT failed login attempts" >> $LOG_FILE
    echo "Warning – $FAILED_COUNT failed login attempts" | mail -s "Security Alert" [email protected]
  fi
}

check_suspicious_processes() {
  ps aux | awk '$3 > 90 {print}' | while read line; do
    echo "$(date): High CPU process: $line" >> $LOG_FILE
  done
}

check_failed_logins
check_suspicious_processes
# Add to crontab: */5 * * * * /path/to/security_monitor.sh

4.3 Intrusion Detection System (OSSEC) Example

<ossec_config>
  <rules>
    <include>rules_config.xml</include>
    <include>sshd_rules.xml</include>
    <include>web_rules.xml</include>
  </rules>
  <syscheck>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
  </syscheck>
</ossec_config>

5. Response Checklist

5.1 Detection Phase ✅

Record detection time and alert details.

Preliminarily assess impact scope.

Notify the security team.

Begin documenting the handling process.

5.2 Confirmation Phase ✅

Verify the intrusion actually occurred.

Identify the attack type.

Assess the affected assets.

Determine the attack path.

5.3 Containment Phase ✅

Isolate compromised systems.

Terminate malicious activity.

Secure critical data.

Collect forensic evidence.

5.4 Recovery Phase ✅

Remove malicious files.

Patch identified vulnerabilities.

Restore normal services.

Validate system security.

5.5 Summary Phase ✅

Write an incident report.

Analyze root cause.

Improve security policies.

Update the emergency response plan.

6. Practical Tool Recommendations

6.1 Detection Tools

chkrootkit

– rootkit scanner rkhunter – system security scanner lynis – security audit aide – file integrity checker

6.2 Monitoring Tools

osquery

– OS instrumentation OSSEC – host‑based IDS Suricata – network IDS ELK Stack – log aggregation and analysis

6.3 Incident‑Response Tools

volatility

– memory forensics foremost – file recovery tcpdump – packet capture strace – system‑call tracing

7. Experience & Best Practices

7.1 Preventive Over Reactive

Regular OS updates and patch management.

Comprehensive monitoring and alerting.

Detailed incident‑response playbooks with periodic drills.

7.2 Log Management Is Crucial

Centralize logs for easier analysis.

Retain sufficient log history.

Define analysis rules and alerts.

7.3 Automation Is the Future

Automated security‑check scripts.

Automated response workflows.

Automated system hardening actions.

7.4 Common Pitfalls to Avoid

Reinstalling the system without preserving evidence.

Cleaning malicious files before forensic collection.

Relying solely on a firewall for protection.

Perform forensic collection first, then clean.

Analyze the root cause to prevent recurrence.

Implement layered, defense‑in‑depth security.

Conclusion

Linux server security is an ongoing process that requires diligent monitoring, well‑defined response procedures, and continuous improvement. By establishing robust detection mechanisms, maintaining up‑to‑date defenses, and practicing regular incident drills, you can minimize damage and recover swiftly from security events.