Master Linux Intrusion Detection & Response: A Complete Hands‑On Guide

Preface: As a sysadmin, you may receive an alert at 3 am—abnormal server behavior, traffic spikes, CPU surges—signs of a possible intrusion. This guide provides a complete Linux server intrusion detection and emergency response workflow to help you stay organized, quickly locate, and resolve issues.

Intrusion Detection: Early Discovery and Handling

1. System Anomaly Metric Monitoring

Key monitoring indicators list:

# System load anomaly detection
uptime # view system load
top -bn1 | head -20 # CPU and memory usage

# Network connection anomaly detection
netstat -tuln | grep LISTEN # listening ports
ss -tulnp | grep :22 # SSH port connection status
netstat -an | grep ESTABLISHED | wc -l # current connections

# Process anomaly detection
ps aux --sort=-%cpu | head -10 # top CPU processes
ps aux --sort=-%mem | head -10 # top memory processes

Practical monitoring script:

#!/bin/bash
# security_check.sh - quick security check script
echo "=== System Security Check Report $(date) ==="

# 1. Check suspicious processes
echo "== Suspicious Process Check =="
ps aux | awk '$3 > 80 {print "High CPU process: " $0}'
ps aux | awk '$4 > 80 {print "High memory process: " $0}'

# 2. Check abnormal network connections
echo "== Network Connection Check =="
netstat -an | grep ":80 " | wc -l | awk '{print "HTTP connections: " $1}'
netstat -an | grep ":22 " | wc -l | awk '{print "SSH connections: " $1}'

# 3. Check login security
echo "== Login Security Check =="
last -n 10 | grep -v "reboot\|shutdown"
lastb | head -5 # failed login attempts

2. Log Analysis and Anomaly Identification

Core log file monitoring:

# Authentication log analysis
tail -f /var/log/auth.log | grep "Failed password" # brute‑force detection
grep "Accepted password" /var/log/auth.log | tail -10 # successful login records

# System log analysis
tail -f /var/log/syslog | grep -E "(error|warning|critical)"

# Web access log analysis (Nginx example)
tail -f /var/log/nginx/access.log | awk '$9 ~ /^4|^5/ {print $0}' # 4xx/5xx errors

One‑click log analysis script:

#!/bin/bash
# log_analyzer.sh - log anomaly analysis
echo "=== Log Security Analysis $(date) ==="

# SSH brute‑force detection
echo "== SSH Brute‑Force Attempts =="
grep "Failed password" /var/log/auth.log | tail -20 | awk '{print $11, $13}' | sort | uniq -c | sort -nr

# Sudo usage records
echo "== Sudo Usage Records =="
grep "sudo:" /var/log/auth.log | tail -10

# New user account changes
echo "== User Account Changes =="
grep "useradd\|userdel" /var/log/auth.log | tail -10

3. File Integrity Check

# Use AIDE for file integrity checking
aide --init   # initialize database
aide --check  # check for file changes

# Manual check of critical system files
find /etc -name "*.conf" -mtime -1 -ls   # config files modified in last day
find /bin /sbin /usr/bin /usr/sbin -perm /u+s -ls   # check SUID files

Intrusion Confirmation: Identify Attack Type

Common Intrusion Feature Identification

1. Webshell detection

# Find suspicious PHP files
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode\|shell_exec" {} \;

# Check recently modified web files
find /var/www -type f -mtime -7 -ls | grep -E "\.(php|jsp|asp)$"

2. Reverse shell detection

# Check suspicious network connections
netstat -antp | grep -E ":4444|:5555|:7777|:8888|:9999"

# Check bash process network connections
lsof -i -a -p `pgrep bash`

3. Mining malware detection

# Detect processes with high CPU usage
top -bn1 | awk 'NR>7 && $9>50 {print "Suspicious process:", $12, "CPU:", $9"%"}'

# Detect known mining process names
ps aux | grep -E "(xmrig|minergate|cpuminer|cgminer)" | grep -v grep

Emergency Response: Rapid Damage Control

Phase 1 – Immediate Response (0‑15 min)

1. Isolate affected systems

# Disconnect network (use with caution)
ifconfig eth0 down

# Or block suspicious IPs
iptables -A INPUT -s [suspicious IP] -j DROP
iptables -A OUTPUT -d [suspicious IP] -j DROP

2. Terminate malicious processes

# Kill process by PID
kill -9 [malicious PID]

# Batch kill by name
pkill -f [malicious process name]

# Terminate suspicious user sessions
pkill -u [suspicious username]

3. Protect critical data

# Backup important logs
cp /var/log/auth.log /tmp/auth.log.backup.$(date +%Y%m%d_%H%M%S)
cp /var/log/syslog /tmp/syslog.backup.$(date +%Y%m%d_%H%M%S)

# Backup configuration
tar -czf /tmp/config_backup_$(date +%Y%m%d_%H%M%S).tar.gz /etc/

Phase 2 – In‑Depth Investigation (15‑60 min)

1. Attack path analysis

#!/bin/bash
# incident_analysis.sh - intrusion path analysis
echo "=== Intrusion Path Analysis $(date) ==="

# Attack timeline
echo "== Attack Timeline =="
grep -E "(Failed|Accepted)" /var/log/auth.log | tail -50

# File modification timeline
echo "== File Modification Timeline =="
find /var/www /tmp /etc -type f -mtime -1 -exec ls -la {} \; | sort -k6,7

2. Vulnerability analysis and confirmation

# Check for upgradable packages
apt list --upgradable   # Debian/Ubuntu
yum check-update       # CentOS/RHEL

# Check service versions
nginx -v
apache2 -v
mysql --version
php -v

Phase 3 – System Hardening (1‑4 h)

1. Clean malicious files

# Delete Webshell files
find /var/www -name "*.php" -exec grep -l "eval\|base64_decode" {} \; | xargs rm -f

# Clean temporary directories
find /tmp -type f -mtime -1 -exec file {} \; | grep -E "(script|executable)"

2. Account security hardening

# Lock suspicious accounts
usermod -L [suspicious username]

# Enforce password policy
chage -M 90 -m 1 -W 7 [username]

# Check SSH keys
cat ~/.ssh/authorized_keys

3. Firewall rule optimization

# Basic firewall configuration
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -j DROP

# Save rules
iptables-save > /etc/iptables/rules.v4

Prevention Hardening: Build a Security Defense

1. Monitoring and Alert System Setup

Use Prometheus + Grafana monitoring solution:

# prometheus.yml example
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

rule_files:
  - "security_rules.yml"

# security_rules.yml
groups:
  - name: security_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"

2. Automated detection scripts

#!/bin/bash
# security_monitor.sh - automatic security monitoring script
LOG_FILE="/var/log/security_monitor.log"

check_failed_logins() {
    FAILED_COUNT=$(grep "Failed password" /var/log/auth.log | grep "$(date '+%b %d')" | wc -l)
    if [ $FAILED_COUNT -gt 10 ]; then
        echo "$(date): Warning - $FAILED_COUNT failed login attempts" >> $LOG_FILE
        echo "Detected abnormal login attempts" | mail -s "Security Alert" [email protected]
    fi
}

check_suspicious_processes() {
    ps aux | awk '$3 > 90 {print $0}' | while read line; do
        echo "$(date): High CPU process: $line" >> $LOG_FILE
    done
}

# Add to crontab: */5 * * * * /path/to/security_monitor.sh
check_failed_logins
check_suspicious_processes

3. Intrusion Detection System Configuration

OSSEC configuration example:

<ossec_config>
  <rules>
    <include>rules_config.xml</include>
    <include>sshd_rules.xml</include>
    <include>web_rules.xml</include>
  </rules>

  <syscheck>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
  </syscheck>
</ossec_config>

Emergency Response Checklist

Discovery Phase ✅

Record discovery time and alert information

Preliminarily assess impact scope

Notify relevant security team

Start documenting handling process

Confirmation Phase ✅

Confirm intrusion facts

Identify attack type

Assess impact scope

Determine attack path

Disposition Phase ✅

Isolate affected systems

Terminate malicious activity

Protect critical data

Collect evidence

Recovery Phase ✅

Clean malicious files

Patch security vulnerabilities

Restore normal services

Verify system security

Summary Phase ✅

Write incident report

Analyze root cause

Optimize security policies

Update emergency plan

Conclusion

Linux server security is an ongoing process that requires preparation in prevention, detection, response, and recovery. By establishing a complete monitoring system, detailed emergency plans, and regular security drills, you can respond quickly to incidents and minimize loss.

Remember, security is not a one‑time effort but a continuous improvement cycle. This guide aims to help you build a robust Linux server protection framework.