Master Linux Server Intrusion Detection & Response: A Complete Practical Guide

Intrusion Detection: Early Discovery

1. System Anomaly Monitoring

Key metrics to watch include system load, CPU and memory usage, network connections, and suspicious processes. Example commands:

# System load detection
uptime
# Top 20 processes by CPU and memory
top -bn1 | head -20
# Network listening ports
netstat -tuln | grep LISTEN
ss -tulnp | grep :22
netstat -an | grep ESTABLISHED | wc -l
# Top CPU‑intensive processes
ps aux --sort=-%cpu | head -10
# Top memory‑intensive processes
ps aux --sort=-%mem | head -10

A ready‑to‑run security script ( security_check.sh) prints a concise report and checks for high‑CPU or high‑memory processes, abnormal network connections, and recent failed logins.

#!/bin/bash
# security_check.sh - Quick security check

echo "=== System Security Report $(date) ==="

echo "== Suspicious Process Check =="
ps aux | awk '$3 > 80 {print "High CPU process: " $0}'
ps aux | awk '$4 > 80 {print "High memory process: " $0}'

echo "== Network Connection Check =="
netstat -an | grep ":80 " | wc -l | awk '{print "HTTP connections: " $1}'
netstat -an | grep ":22 " | wc -l | awk '{print "SSH connections: " $1}'

echo "== Login Security Check =="
last -n 10 | grep -v "reboot\|shutdown"
lastb | head -5

2. Log Analysis & Anomaly Identification

Monitor authentication logs for brute‑force attempts and system logs for errors. Example one‑liner scripts:

# Auth log tail for failed passwords
tail -f /var/log/auth.log | grep "Failed password"
# Successful logins
grep "Accepted password" /var/log/auth.log | tail -10
# System log errors and warnings
tail -f /var/log/syslog | grep -E "(error|warning|critical)"
# Nginx access log for 4xx/5xx responses
tail -f /var/log/nginx/access.log | awk '$9 ~ /^4|^5/ {print $0}'

A script ( log_analyzer.sh) aggregates SSH brute‑force attempts, sudo usage, and user account changes.

#!/bin/bash
# log_analyzer.sh - Log anomaly analysis

echo "=== Log Security Analysis $(date) ==="

echo "== SSH brute‑force attempts =="
grep "Failed password" /var/log/auth.log | tail -20 | awk '{print $11, $13}' | sort | uniq -c | sort -nr

echo "== Sudo usage =="
grep "sudo:" /var/log/auth.log | tail -10

echo "== User account changes =="
grep "useradd\|userdel" /var/log/auth.log | tail -10

3. File Integrity Verification

Use AIDE for automated integrity checks and manual find commands for recent configuration changes.

# AIDE initialization and check
aide --init   # creates database
aide --check  # compares current files
# Find recently modified config files (last 1 day)
find /etc -name "*.conf" -mtime -1 -ls
# List SUID binaries
find /bin /sbin /usr/bin /usr/sbin -perm /u+s -ls

Intrusion Confirmation: Identify Attack Type

Webshell Detection

# Find suspicious PHP files
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode\|shell_exec" {} \;
# List recently modified web files (last 7 days)
find /var/www -type f -mtime -7 -ls | grep -E "\.(php|jsp|asp)$"

Reverse Shell Detection

# Check for common reverse‑shell ports
netstat -antp | grep -E ":4444|:5555|:7777|:8888|:9999"
# Inspect bash processes for network sockets
lsof -i -a -p `pgrep bash`

Mining Malware Detection

# High‑CPU processes that may be miners
top -bn1 | awk 'NR>7 && $9>50 {print "Suspicious process:", $12, "CPU:", $9"%"}'
# Known miner process names
ps aux | grep -E "(xmrig|minergate|cpuminer|cgminer)" | grep -v grep

Emergency Response: Rapid Containment

Phase 1 – Immediate Response (0‑15 min)

1. Isolate the affected system

# Disconnect network (use with caution)
ifconfig eth0 down
# Or block suspicious IPs
iptables -A INPUT -s [SUSPICIOUS_IP] -j DROP
iptables -A OUTPUT -d [SUSPICIOUS_IP] -j DROP

2. Terminate malicious processes

# Kill by PID
kill -9 [MALICIOUS_PID]
# Kill by name pattern
pkill -f [MALICIOUS_NAME]
# End suspicious user sessions
pkill -u [SUSPICIOUS_USER]

3. Protect critical data

# Backup important logs
cp /var/log/auth.log /tmp/auth.log.backup.$(date +%Y%m%d_%H%M%S)
cp /var/log/syslog /tmp/syslog.backup.$(date +%Y%m%d_%H%M%S)
# Archive configuration files
tar -czf /tmp/config_backup_$(date +%Y%m%d_%H%M%S).tar.gz /etc/

Phase 2 – Deep Investigation (15‑60 min)

1. Attack path analysis

#!/bin/bash
# incident_analysis.sh - Intrusion path analysis

echo "=== Intrusion Path Analysis $(date) ==="

echo "== Attack timeline =="
grep -E "(Failed|Accepted)" /var/log/auth.log | tail -50

echo "== File modification timeline =="
find /var/www /tmp /etc -type f -mtime -1 -exec ls -la {} \; | sort -k6,7

echo "== Process analysis =="
ps -eo pid,ppid,cmd,etime,user --sort=etime

2. Vulnerability verification

# List upgradable packages (Debian/Ubuntu)
apt list --upgradable
# List upgradable packages (CentOS/RHEL)
yum check-update
# Show versions of common services
nginx -v
apache2 -v
mysql --version
php -v

Phase 3 – System Hardening (1‑4 h)

1. Clean malicious files

# Remove webshells
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode" {} \; | xargs rm -f
# Clean temporary directories
find /tmp -type f -mtime -1 -exec file {} \; | grep -E "(script|executable)"

2. Account security

# Lock suspicious accounts
usermod -L [SUSPICIOUS_USER]
# Enforce password policy
chage -M 90 -m 1 -W 7 [USERNAME]
# Review SSH authorized keys
cat ~/.ssh/authorized_keys

3. Firewall rule optimization

# Basic iptables configuration
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -j DROP
# Save rules
iptables-save > /etc/iptables/rules.v4

Prevention Hardening: Building a Security Perimeter

1. Monitoring & Alerting Stack

Deploy Prometheus + Grafana to collect node metrics and trigger alerts for abnormal CPU usage.

# prometheus.yml (excerpt)
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']
rule_files:
  - "security_rules.yml"

# security_rules.yml (excerpt)
groups:
  - name: security_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"

2. Automated Detection Scripts

#!/bin/bash
# security_monitor.sh - Automated security monitor
LOG_FILE="/var/log/security_monitor.log"

check_failed_logins() {
  FAILED_COUNT=$(grep "Failed password" /var/log/auth.log | grep "$(date '+%b %d')" | wc -l)
  if [ $FAILED_COUNT -gt 10 ]; then
    echo "$(date): Warning - $FAILED_COUNT failed login attempts" >> $LOG_FILE
    echo "Detection of abnormal login attempts" | mail -s "Security Alert" [email protected]
  fi
}

check_suspicious_processes() {
  ps aux | awk '$3 > 90 {print}' | while read line; do
    echo "$(date): High CPU process: $line" >> $LOG_FILE
  done
}

# Add to crontab: */5 * * * * /path/to/security_monitor.sh
check_failed_logins
check_suspicious_processes

3. Intrusion Detection System (OSSEC) Example

<ossec_config>
  <rules>
    <include>rules_config.xml</include>
    <include>sshd_rules.xml</include>
    <include>web_rules.xml</include>
  </rules>
  <syscheck>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
  </syscheck>
</ossec_config>

Response Checklist

Discovery Phase

Record detection time and alert details.

Assess initial impact scope.

Notify the security team.

Begin documenting remediation steps.

Confirmation Phase

Verify intrusion occurrence.

Identify attack type.

Evaluate impact range.

Determine attack path.

Containment Phase

Isolate affected hosts.

Terminate malicious activity.

Secure critical data.

Collect forensic evidence.

Recovery Phase

Remove malicious files.

Patch identified vulnerabilities.

Restore normal services.

Validate system integrity.

Post‑Incident Review

Write a detailed incident report.

Analyze root cause.

Improve security policies.

Update the emergency response plan.

Best Practices & Experience

Key Takeaways

Regular system updates and patch management are essential.

Maintain a comprehensive monitoring and alerting system.

Develop and rehearse detailed incident response procedures.

Log data is invaluable; centralize logs and retain sufficient history.

Automation reduces response time and human error.

Avoid common pitfalls such as immediate system re‑installation without evidence collection.

Adopt layered defenses for depth security.

By treating security as a continuous process—covering prevention, detection, response, and recovery—administrators can minimize damage and restore services swiftly when a Linux server is compromised.