Linux Intrusion Detection and Incident Response: A Practical Guide to Security Event Investigation

1. Overview

Production environments cannot rely solely on perimeter firewalls; once an attacker breaches the outer layer, host‑level detection is essential to catch lateral movement, privilege escalation, and backdoors before business impact or data loss occurs.

The intrusion detection ecosystem is divided into three layers:

Network layer (NIDS) : traffic analysis with tools like Suricata or Snort – good for known signatures but ineffective on encrypted traffic.

Host layer (HIDS) : runs on the target host, monitors file changes, process behavior, and user actions – the final line of defense.

Application layer : WAF, RASP, etc., protect web applications.

Host‑based detection focuses on five capabilities: file integrity, rootkit scanning, log auditing, process monitoring, and network connection anomaly detection.

2. Tool Comparison

Common HIDS tools are compared on file integrity, rootkit detection, real‑time monitoring, centralized management, and deployment complexity. No single tool covers all scenarios; a recommended combination is AIDE for baseline file integrity, rkhunter for periodic rootkit scans, auditd for kernel‑level audit, and ClamAV for malware scanning.

3. Environment Requirements

Operating System: Ubuntu 22.04+ or CentOS 8+ (Ubuntu 22.04 LTS preferred).

AIDE version 0.18.x (install from default apt repository).

rkhunter 1.4.x (update signature database after installation).

auditd 3.x (requires kernel 4.x+ for full features).

ClamAV 1.x (keep virus database up‑to‑date with freshclam).

Disk space: at least 10 GB for audit logs.

4. Detailed Steps

4.1 File Integrity Monitoring with AIDE

Install AIDE:

# Ubuntu/Debian
sudo apt update && sudo apt install -y aide

# CentOS/RHEL
sudo dnf install -y aide

Configure /etc/aide/aide.conf to define rule groups (e.g., BINLIB, CONFFILE, LOGFILE) and exclude frequently changing directories ( /var/log, /proc, etc.).

Initialize the baseline database and store it on read‑only media:

# Initialize baseline
sudo aide --init
# Promote new database to active baseline
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# Secure copy to offline storage
sudo cp /var/lib/aide/aide.db /mnt/readonly-backup/aide.db.$(date +%Y%m%d)

Schedule daily checks via cron:

# /etc/cron.d/aide-check
0 3 * * * root /usr/bin/aide --check | mail -s "AIDE Report $(hostname) $(date +%F)" [email protected]

4.2 Rootkit Scanning with rkhunter (and chkrootkit)

Install and update signatures:

sudo apt install -y rkhunter
sudo rkhunter --update
sudo rkhunter --propupd

Key configuration in /etc/rkhunter.conf includes enabling automatic updates, setting email alerts, whitelisting known false‑positives, and defining the package manager.

Run a full scan:

# Interactive scan
sudo rkhunter --check
# Non‑interactive for automation
sudo rkhunter --check --skip-keypress --report-warnings-only

Cross‑validate with chkrootkit to reduce false negatives.

4.3 System Auditing with auditd

Install auditd and start it:

sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd

Load a comprehensive rule set (e.g., /etc/audit/rules.d/99-security-hardening.rules) that covers time changes, identity modifications, privilege escalation, network configuration, SSH settings, cron persistence, kernel module loading, file deletions, and temporary‑directory execution. End the file with -e 2 to lock the rules.

Query logs with ausearch and generate reports with aureport (e.g., ausearch -k privilege_escalation -ts recent).

4.4 Incident Response Process

The six‑step response workflow:

Preparation : toolchain ready, response plan, contact list.

Identification : confirm the event, assess scope.

Containment : block attacker (network isolation, firewall rules) and apply short‑term and long‑term mitigations.

Eradication : remove backdoors, malicious files, and compromised accounts.

Recovery : restore from trusted backups, bring services back online.

Lessons Learned : post‑mortem analysis, update defenses, revise playbooks.

Initial response checklist includes gathering system info ( uname -a, OS release), user account audit, process inspection, network connection review, recent file changes, SUID/SGID discovery, and cron job enumeration.

4.5 Automation Scripts

A unified security audit script ( security_audit.sh) collects hostname, IP, OS version, kernel version, and runs checks for users, processes, network sockets, crontabs, and SUID/SGID files, producing per‑section reports and a tarball.

An intrusion‑trace script ( intrusion_detect.sh) flags abnormal users, permission anomalies, hidden processes, deleted executables, suspicious network connections, webshells, mining processes, and high‑CPU usage, summarizing alerts at the end.

A mining‑removal script demonstrates detection of high‑CPU miners, termination of malicious processes, deletion of associated files, and cleanup of persistence mechanisms (crontab, systemd services).

4.6 Best Practices and Caveats

Store AIDE baseline databases offline; otherwise attackers can regenerate a clean baseline.

Keep auditd rule count reasonable (≈50 rules) to avoid performance impact.

Never power‑off a compromised host before collecting volatile data; memory, network state, and /proc information are critical evidence.

Prefer imaging the disk before any cleanup; use dd or LVM snapshots for forensics.

Rotate SSH keys and enforce “PermitRootLogin no”.

Apply CIS Benchmark hardening and remove unnecessary services.

Use immutable file attributes (e.g., chattr +a /var/log/auth.log) and remote log forwarding (TCP) to protect logs.

5. Monitoring and Metrics

Key security metrics to collect continuously include SSH failed login count, SUID file count changes, abnormal outbound connections, high‑CPU processes, executable files in /tmp, and new crontab entries. Example Prometheus exporter script writes these metrics to /var/lib/prometheus/node-exporter/security_metrics.prom for alerting.

Prometheus alert rules trigger on thresholds such as >20 SSH failures in 5 minutes, any change in SUID count, presence of non‑standard outbound connections, or executable files in temporary directories.

6. Backup, Forensics, and Recovery

Immediately create a disk image with dd (e.g.,

dd if=/dev/sda of=/mnt/forensic/disk_image_$(date +%Y%m%d_%H%M%S).raw bs=4M conv=noerror,sync status=progress

) and compute a SHA‑256 hash for integrity verification.

For live systems using LVM, create a read‑only snapshot, mount it, and collect evidence without disrupting production.

Standard post‑incident recovery steps:

Assess impact and scope of compromise.

Isolate the host (network segmentation, firewall drops) while preserving evidence.

Remove malicious files and persistence mechanisms (crontab, systemd units, rc.local, authorized_keys).

Patch exploited vulnerabilities and harden configurations.

Reinstall the OS from a trusted image if the compromise is severe.

Re‑deploy applications from clean sources.

Implement HIDS, remote logging, network segmentation, and rotate all credentials.

Document the incident, update response playbooks, and conduct security awareness training.

7. Further Learning

Deploy Wazuh for enterprise‑grade HIDS with centralized management and active response.

Explore eBPF‑based detection with Falco or Tetragon for kernel‑level visibility.

Map detection rules to the MITRE ATT&CK framework to identify coverage gaps.

8. References

CIS Benchmarks – OS hardening guides.

AIDE official documentation.

Linux Audit System – Red Hat documentation.

RFC 3227 – Digital forensics evidence collection priorities.

Wazuh documentation – Open‑source HIDS platform.