How I Recovered a Compromised Linux Server: Step‑by‑Step Incident Response

1. Signs of Server Compromise

A friend’s website server suddenly showed 100% CPU usage, high load, and services became unavailable, indicating a possible intrusion.

2. Investigation and Remediation

2.1 Possible Causes

Weak SSH password.

Overly permissive cloud security group.

Simple password for the Baota control panel.

2.2 Investigation Steps

Identify the most resource‑intensive processes: ps -ef / top Check login records: last or

grep 'Accepted' /var/log/secure

[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
...

Observe foreign IPs (e.g., 34.215.138.2) successfully logging in.

2.3 Immediate Measures

Restrict SSH access in the cloud security group to specific IPs.

Change the root SSH password.

Backup and clear /root/.ssh/authorized_keys.

Attempt to copy /root/.ssh/authorized_keys but encounter permission errors, indicating locked files.

2.4 User Account Review

Lock suspicious users:

[root@VM-12-12-centos ~]# usermod -L sys1

2.5 Cron Job and Startup Script Analysis

Inspect cron directories ( /var/spool/cron/, /etc/crontab, /etc/cron.d/) and logs ( /var/log/cron) for hidden tasks. Example log entries show recurring root cron jobs executing unknown binaries such as /sbin/httpss and /usr/local/qcloud/YunJing/YDCrontab.sh.

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1)

Examine /etc/rc.local which contains a suspicious command /usr/bin/0f4f80f9ab start, then comment it out.

2.6 Restoring Replaced System Binaries

Commands like ps, top, chattr, and lsattr were replaced and locked. Copy the originals from an identical machine to /tmp, then unlock and replace:

/tmp/chattr -ai /usr/bin/chattr

/tmp/chattr -ai /usr/bin/

After unlocking, replace the compromised binaries.

2.7 Removing Malicious Scripts

A persistent script located at /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 continuously kills security tools and reinstalls a malicious shared library via /etc/ld.so.preload. Kill the process, delete the script, and remove the preload entry.

#!/bin/sh
while true; do
  sleep 30
  pkill -f main
  ...
  echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
  chattr +ai /etc/ld.so.preload
  ...
done

After deleting the script, the /etc/ld.so.preload file was cleared, but the malicious entry reappeared, indicating a hidden cron job.

3. Lessons Learned

Use restrictive cloud security group rules for critical ports.

Enforce strong passwords for SSH and control panels.

Monitor critical files (e.g., binaries, /etc/ld.so.preload) with integrity checks such as MD5 hashes.

When a server is compromised, isolate it, restore system binaries from a clean source, and thoroughly examine cron jobs, startup scripts, and preload configurations.

During Linux dynamic linking, the loader reads LD_PRELOAD and /etc/ld.so.preload , loading any specified libraries before normal ones, which can be abused by attackers to inject malicious code.

References:

https://cloud.tencent.com/document/product/296/9604

https://help.aliyun.com/document_detail/40994.htm