How to Detect and Recover from a Compromised Linux Server: Step‑by‑Step Guide

下文中的，给文件和目录加锁，是指给文件和目录增加了一些属性，只读等。chattr +ia

1. Server intrusion symptoms

Recently a friend's server hosting a website was compromised, showing sustained 100% CPU usage, high load, and services failing.

2. Investigation and remediation

2.1 Possible causes of intrusion

Simple SSH password.

Broad security group rules.

Easy password for the BaoTa panel.

2.2 Investigation steps

1. Use ps -ef / top to find the process consuming most resources.

Problem: ps/top command had been replaced.

2. Search logs with last or grep 'Accepted' /var/log/secure.

[root@VM-12-12-centos ~]# grep 'Accepted'  /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
Aug 27 08:52:05 VM-12-12-centos sshd[3053]: Accepted password for root from 127.0.0.1 port 57534 ssh2
... (additional log lines) ...

Found foreign IP 34.215.138.2 successfully logged in many times.

Remediation measures

Restrict SSH login IPs in the cloud security group.

Change the root SSH password.

Backup and clear /root/.ssh/authorized_keys.

# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied

Encountered permission issues when trying to modify files.

3. Check newly added users with cat /etc/passwd.

Remediation

Lock the suspicious user.

# usermod -L sys1

4. Examine scheduled tasks; many cron locations were empty, but logs showed recurring execution of unknown scripts.

/var/spool/cron/ (not found)

/etc/crontab (not found)

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1)
Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)
... (additional cron lines) ...

Removed suspicious binaries /usr/lib/mysql/mysql and /sbin/httpss, but chattr itself was locked.

Found a persistent script in /etc/rc.local that repeatedly modifies /etc/ld.so.preload and kills security tools.

# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
...
/usr/bin/0f4f80f9ab start

Deleted the script after killing its process.

3. Lessons learned

Key points:

Commands like ps, top, chattr, lsattr can be replaced; keep copies on another machine for recovery.

Malicious files may hide their content using special attributes.

Persistent scripts can modify /etc/ld.so.preload to load malicious libraries.

4. Recommendations

Use cloud security groups to restrict critical ports.

Strengthen passwords for SSH and management panels.

Monitor critical files (e.g., /etc/passwd, /etc/shadow, /etc/ld.so.preload) for changes.

Reference: https://cloud.tencent.com/document/product/296/9604 https://help.aliyun.com/document_detail/40994.htm?spm=a2c4g.11186623.0.0.75c56956NVPBST