When fail2ban Became a Monero Miner: Detection, Removal, and Prevention

Before the Chinese New Year, a project required two short‑term test servers on Tianyi Cloud for a low‑altitude drone platform. Because the machines were temporary, they were not added to the monitoring system and security measures were minimal.

Unexpected CPU Spike

During a routine check the top command showed a process named fail2ban consuming 1596% CPU, which is abnormal since fail2ban normally uses negligible resources.

Investigation Reveals a Hidden Miner

Using netstat -anlp | grep fail2ban the author found a persistent connection to 154.204.228.130:44444, an address belonging to a cryptocurrency mining pool. Log entries indicated the process attempted to modify CPU registers ("FAILED TO APPLY MSR MOD") to boost hash rate.

The configuration file showed that the process was not the genuine fail2ban service but an XMRig miner disguised under the fail2ban name, silently mining Monero.

Emergency Remediation Steps

Terminate the malicious process : kill -9 2093165 Remove malicious files : delete the fake fail2ban binary and related configuration files.

Check for backdoors : inspect crontab, systemd services, SSH keys, etc., to ensure no persistence mechanisms remain.

Restore legitimate services : reinstall the authentic fail2ban package and start it.

Strengthen the system :

Restrict SSH login IPs, disable password authentication, enforce key‑based login.

Install a process‑monitoring tool to catch abnormal CPU usage early.

Regularly update the OS and software to patch vulnerabilities.

Quick Diagnostic Commands (copy‑and‑paste)

# Verify the real binary of the process
ls -l /proc/$(pgrep fail2ban 2>/dev/null || echo 1)/exe

# Search for XMRig signatures
strings $(which fail2ban-server 2>/dev/null || echo /tmp/fake) | grep -iE "xmr|monero|pool|hashrate"

# Scan for suspicious outbound connections
netstat -anp | grep -E ":(3333|4444|5555|44444)" | grep ESTABLISHED

Rapid 5‑Minute Damage Control

# Kill the malicious process
pkill -9 -f "fail2ban.*golden|fail2ban.*44444"

# Remove disguised files (use with caution)
rm -f /etc/fail2ban/fail2ban /tmp/.X11-unix/fail2ban /dev/shm/.fail2ban

# Block the mining pool IPs
iptables -A OUTPUT -d 154.204.228.130 -j DROP
iptables -A OUTPUT -d $(dig +short goldenroad.top) -j DROP

Root‑Cause Elimination

# Check for persistence entries
crontab -l | grep -i fail2ban && crontab -r
systemctl list-unit-files | grep fail2ban | xargs -I{} systemctl disable {}

# Full system scan
apt install rkhunter chkrootkit -y
rkhunter --check --sk
chkrootkit

Rebuilding Protection

# Reinstall genuine fail2ban
apt purge fail2ban -y
apt install fail2ban -y
systemctl enable --now fail2ban

# Harden SSH configuration
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Preventive Checklist

Never trust a process name; verify the executable path with ps -ef | grep [process] and ls -l /proc/[PID]/exe.

Monitor network connections regularly using netstat or ss and investigate unknown IPs/ports.

Limit execution permissions on temporary directories such as /tmp and /var/tmp.

Enable system auditing (e.g., auditd) to track file modifications and process creation.

Adopt daily health checks: review load average, process list, and logs.

Even short‑lived test machines must have basic firewall rules and SSH hardening to avoid becoming a “zombie” host.

The incident underscores that security cannot be an afterthought, even for temporary servers; vigilant monitoring, rapid forensics, and solid hardening are essential to protect infrastructure from stealthy mining trojans.