Server Intrusion Investigation and Remediation Steps

Recently a friend's website server was compromised, showing sustained 100% CPU usage and high load, rendering services unavailable.

The investigation began by examining possible causes: weak SSH password, overly permissive cloud security group, and simple passwords on the Baota panel.

Investigation steps:

Identify resource‑heavy processes using ps -ef and top . The commands themselves had been replaced.

Search login records with grep 'Accepted' /var/log/secure to find foreign IPs (e.g., 34.215.138.2) that successfully logged in.

Check recent user additions via cat /etc/passwd .

Inspect cron jobs and startup scripts ( /etc/rc.local , /etc/crontab , /var/spool/cron ) and notice hidden or corrupted files.

Examine /etc/ld.so.preload for malicious library preload entries (e.g., /usr/local/lib/libprocesshider.so ).

# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
...

A persistent malicious script was found under /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 , which continuously kills security‑related processes and rewrites /etc/ld.so.preload .

# cat /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464
#!/bin/sh
while test 1 = 1
do
  sleep 30
  pkill -f main
  ...
  echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
  lockr +ai /etc/ld.so.preload >/dev/null 2>&1
  ...
done

Remediation actions:

Restrict SSH access to specific IPs via the cloud security group.

Change the root password and backup then clear /root/.ssh/authorized_keys .

Unlock and replace locked binaries (e.g., chattr -ai /usr/bin/chattr , copy original utilities from a clean machine, and restore top , ps , lsattr ).

Delete malicious files and scripts, stop related services, and clear /etc/ld.so.preload .

Lock newly created users using usermod -L sys1 .

# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak

Key lessons learned:

Use minimal security‑group rules and restrict critical ports.

Enforce strong, complex passwords for all accounts.

Monitor critical files (e.g., /etc/passwd , /etc/shadow , /etc/ld.so.preload ) and binaries for unauthorized changes.

Regularly audit cron jobs, startup scripts, and running processes.

When files become immutable, use chattr -ia after restoring a clean copy.

For further reference, see the original article at cnblogs.com/operationhome/p/16637763.html .