How to Investigate and Harden a Compromised Linux Server: Real-World Case Study

Overview

A production Linux server experienced a sudden CPU spike to 100 % caused by a mining malware that entered through an unauthenticated Redis instance. The article documents the full response workflow, from initial triage to long‑term hardening.

Technical Characteristics

Time‑sensitive : Attackers may still be active, so rapid detection is critical.

Systematic investigation required : Processes, network, users, cron jobs, and startup items must all be examined.

Evidence preservation : Capture snapshots before killing processes or deleting files.

Applicable Scenarios

CPU/Memory spikes suggesting mining.

Suspicious processes or outbound connections.

Security alerts from cloud providers (DDoS, mining, trojan).

Abnormal login records in logs.

Environment Requirements

Operating System: CentOS 7+ or Ubuntu 18.04+

Investigation tools (optional): rkhunter, chkrootkit, unhide

Network tools: netstat/ss, lsof (built‑in)

Detailed Steps

Emergency Response – Stay Calm

Do not immediately kill processes or reinstall the system. First assess business impact and decide whether to isolate the host.

# Check active business connections
ss -ant | grep ESTABLISHED | wc -l
# If isolation is possible:
iptables -I INPUT -j DROP
iptables -I OUTPUT -j DROP
# Keep your own SSH session
iptables -I INPUT -s YOUR_IP -j ACCEPT
iptables -I OUTPUT -d YOUR_IP -j ACCEPT

Investigation Process

Check abnormal processes

# List top CPU consumers
ps aux --sort=-%cpu | head -20
# List top memory consumers
ps aux --sort=-%mem | head -20
# Show full process tree
ps auxf
# Indicators to watch for:
# - Random string names (e.g., aGhsZD)
# - Executables under /tmp, /var/tmp, /dev/shm
# - System‑like names with wrong paths (e.g., /tmp/sshd)

# Examine a suspicious PID (replace 12345 with the actual PID)
ls -la /proc/12345/exe
cat /proc/12345/cmdline | tr '\0' ' '
ls -la /proc/12345/fd/
cat /proc/12345/environ | tr '\0' '
'

Check abnormal network connections

# List all connections (ss is faster than netstat)
ss -antup
# Show only ESTABLISHED connections
ss -antup state established
# Show listening ports
ss -tlnp
# Focus on:
# - Connections to unknown IPs
# - Services listening on unusual ports
# - Many connections to the same external IP

# Using lsof:
lsof -i
lsof -i @EXTERNAL_IP
lsof -p 12345 -i

# Identify IP ownership (requires curl)
curl ip.sb/12.34.56.78
curl ipinfo.io/12.34.56.78

Check abnormal users and privileges

# List all users
cat /etc/passwd
# Find UID 0 users other than root
awk -F: '$3==0 {print $1}' /etc/passwd
# Users with valid shells
grep -v '/nologin\|/false' /etc/passwd
# Recent user creations
ls -lt /home/
# Inspect sudoers
cat /etc/sudoers
ls -la /etc/sudoers.d/
# Check SSH authorized_keys for each user
cat /root/.ssh/authorized_keys
for user in $(ls /home); do
  echo "=== $user ==="
  cat /home/$user/.ssh/authorized_keys 2>/dev/null
done

Check scheduled tasks

# Root crontab
crontab -l
# All users' crontabs
for user in $(cut -d: -f1 /etc/passwd); do
  echo "=== $user ==="
  crontab -l -u $user 2>/dev/null
done
# System‑wide crontabs
cat /etc/crontab
ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
ls -la /etc/cron.weekly/
ls -la /etc/cron.monthly/
# Check anacron
cat /etc/anacrontab
# systemd timers
systemctl list-timers --all

Check startup items

# SysV init scripts
ls -la /etc/init.d/
cat /etc/rc.local
# systemd services
systemctl list-unit-files --type=service | grep enabled
# Recently added/modified services
find /etc/systemd/system -mtime -7 -type f
find /usr/lib/systemd/system -mtime -7 -type f
# User‑level autostart
ls -la ~/.config/autostart/ 2>/dev/null

Check system logs

# Login history
last -n 50
lastb -n 50
# Authentication logs (CentOS/RHEL)
tail -500 /var/log/secure | grep -i 'accepted\|failed'
# Authentication logs (Ubuntu/Debian)
tail -500 /var/log/auth.log | grep -i 'accepted\|failed'
# SSH brute‑force attempts
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20
# General system logs
tail -500 /var/log/messages
journalctl -xe --no-pager | tail -200
# Detect log truncation
ls -la /var/log/

Check for rootkits

# Install and run rkhunter
yum install -y rkhunter   # CentOS
apt install -y rkhunter   # Ubuntu
rkhunter --update
rkhunter --check --sk
# Install and run chkrootkit
yum install -y chkrootkit   # may need EPEL
apt install -y chkrootkit
chkrootkit
# Use unhide to find hidden processes/ports
yum install -y unhide
unhide proc
unhide sys
unhide-tcp

Clean Up Malicious Artifacts

# 1. Stop malicious processes
kill -9 PID
# If a watchdog restarts it, kill the parent as well
ps auxf | grep -A5 malicious_name
# 2. Delete malicious files
rm -f /tmp/malicious_file
rm -f /var/tmp/malicious_file
# 3. Remove suspicious cron entries
crontab -e   # delete lines
rm -f /etc/cron.d/suspicious_file
# 4. Disable and delete malicious services
systemctl disable suspicious_service
rm -f /etc/systemd/system/suspicious_service.service
systemctl daemon-reload
# 5. Delete backdoor users
userdel -r backdoor_user
# 6. Clean unknown SSH keys
vim /root/.ssh/authorized_keys   # remove unknown keys

If a file has the immutable attribute set (chattr +i), remove it before deletion:

# Check file attributes
lsattr /tmp/malicious_file
# Remove immutable flag
chattr -i /tmp/malicious_file
# Delete the file
rm -f /tmp/malicious_file

System Hardening

# 1. Harden SSH
vim /etc/ssh/sshd_config
#   Port 22222
#   PermitRootLogin no
#   PasswordAuthentication no
systemctl restart sshd
# 2. Configure firewall (firewalld example)
firewall-cmd --permanent --add-port=22222/tcp
firewall-cmd --permanent --remove-service=ssh
firewall-cmd --reload
# Or iptables
iptables -A INPUT -p tcp --dport 22222 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP
# 3. Update OS and packages
yum update -y   # CentOS
apt update && apt upgrade -y   # Ubuntu
# 4. Fix the original vulnerability (Redis unauthenticated access)
vim /etc/redis.conf
#   requirepass STRONG_PASSWORD
#   bind 127.0.0.1
# 5. Install fail2ban to block brute‑force attempts
yum install -y fail2ban
systemctl enable fail2ban
systemctl start fail2ban

Example Scripts and Configurations

Full Investigation Script

#!/bin/bash
# security_check.sh – one‑click collection of forensic data

echo "========================================"
echo "Linux Security Investigation Script"
echo "Run time: $(date '+%Y-%m-%d %H:%M:%S')"
echo "Hostname: $(hostname)"
echo "========================================"

# 1. System info
uname -a
cat /etc/os-release 2>/dev/null || cat /etc/redhat-release 2>/dev/null

# 2. Top CPU processes
ps aux --sort=-%cpu | head -11

# 3. Top memory processes
ps aux --sort=-%mem | head -11

# 4. Process tree
ps auxf

# 5. Network connections
ss -antup 2>/dev/null || netstat -antup

# 6. Listening ports
ss -tlnp 2>/dev/null || netstat -tlnp

# 7. UID=0 users
awk -F: '$3==0 {print $1}' /etc/passwd

# 8. Login‑capable users
grep -v '/nologin\|/false' /etc/passwd

# 9. Cron jobs
echo "--- root crontab ---"
crontab -l 2>/dev/null
cat /etc/crontab
ls -la /etc/cron.d/

# 10. SSH authorized keys
echo "--- root ---"
cat /root/.ssh/authorized_keys 2>/dev/null
for user in $(ls /home 2>/dev/null); do
  echo "--- $user ---"
  cat /home/$user/.ssh/authorized_keys 2>/dev/null
done

# 11. Recent logins
last -n 20

# 12. Failed logins
lastb -n 20 2>/dev/null

# 13. Recent files in /tmp /var/tmp /dev/shm (last 7 days)
find /tmp /var/tmp /dev/shm -mtime -7 -type f 2>/dev/null

# 14. Executable files in those dirs
find /tmp /var/tmp /dev/shm -type f -executable 2>/dev/null

# 15. Enabled systemd services
systemctl list-unit-files --type=service | grep enabled

# 16. rc.local content
cat /etc/rc.local 2>/dev/null

echo "=== Investigation Completed ==="

Real‑World Cases

Case 1 – Mining Virus

Symptom: CPU at 100 % and a process named kworkerds consuming resources.

# Inspect the process
ps aux | grep kworkerds
# Verify the real executable path
ls -la /proc/12345/exe
# Check network connections of the PID
ss -antp | grep 12345
# Look for malicious cron jobs
crontab -l
# Cleanup
kill -9 12345
rm -rf /tmp/.X11-unix/
crontab -r   # after backup

Root cause: Exploited an old Redis version without authentication, allowing the attacker to write an SSH public key and a malicious cron entry.

Case 2 – SSH Brute‑Force

Symptom: lastb shows many failed logins; a weak‑password account was compromised.

# Identify attacking IPs
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20
# Check successful logins
grep "Accepted" /var/log/secure | tail -20
# Review user history and possible backdoors
cat /home/deploy/.bash_history
cat /home/deploy/.ssh/authorized_keys
crontab -l -u deploy

Hardening actions:

# Disable password authentication
vim /etc/ssh/sshd_config
#   PasswordAuthentication no
# Install and configure fail2ban
yum install -y fail2ban
cat > /etc/fail2ban/jail.local <<'EOF'
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/secure
maxretry = 5
bantime = 3600
EOF
systemctl restart fail2ban

Best Practices and Caveats

Daily Hardening Checklist

SSH security : Change default port, disable root login, enforce key‑based authentication, limit login attempts.

System account lockdown : Set non‑service accounts to /sbin/nologin.

File permissions : chmod 600 /etc/shadow, chmod 644 /etc/passwd, chmod 700 /root.

Immutable critical files : chattr +i /etc/passwd /etc/shadow /etc/sudoers (remove with chattr -i when changes are needed).

Investigation Caveats

Do not rely on binaries on the compromised host; attackers may replace ps, netstat, etc. Use trusted static binaries when possible.

Always preserve evidence (PID, memory dump, network state) before killing processes.

Check other hosts in the network – lateral movement is common.

When to Reinstall

Rootkits detected.

Kernel modules have been tampered with.

Forensic effort exceeds the cost of a clean reinstall.

Unable to verify complete cleanup.

Monitoring and Alerting

Key Metrics

CPU usage : Normal 0‑70 %; alert if >90 % for 5 minutes.

Abnormal outbound connections : Normal <50/min; alert if >200/min.

SSH login failures : Normal <10/hour; alert if >100/hour.

New user accounts : Any creation should trigger an alert.

Cron changes : Any addition/modification should trigger an alert.

Simple Alert Script (run via cron every minute)

#!/bin/bash
# Alert on high CPU
cpu=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d% -f1)
if (( $(echo "$cpu > 90" | bc -l) )); then
  echo "Alert: CPU usage $cpu%"
fi
# Alert on connections to common mining ports
suspicious_ports="3333 4444 5555 6666 7777"
for port in $suspicious_ports; do
  count=$(ss -ant | grep ":$port" | grep ESTAB | wc -l)
  if (( count > 0 )); then
    echo "Alert: Connection to suspicious port $port"
  fi
done

Conclusion

Effective incident response starts with staying calm, preserving evidence, and following a systematic checklist that covers processes, network activity, user accounts, scheduled tasks, startup items, and logs. After thorough cleanup, hardening the system—especially SSH, firewall, package updates, and fixing the original vulnerability—prevents recurrence. Regular automated scans and monitoring of the key metrics listed above provide early warning of future compromises.