How to Detect and Recover from a Linux Server Intrusion: A Step‑by‑Step Guide

1. Server intrusion symptoms

A friend’s website server showed sustained 100% CPU usage, high load, and service outages, indicating a possible compromise.

2. Investigation and remediation

2.1 Possible causes

Simple SSH password.

Overly permissive cloud security‑group rules.

Weak password on the Baota control panel.

2.2 Investigation steps

Identify resource‑heavy processes with

ps -ef

or

top

.

Search login records using

last

or

grep 'Accepted' /var/log/secure

to find foreign IPs such as

34.215.138.2

that logged in repeatedly.

Examine

/var/log/secure

for accepted passwords and keys:

<code>[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure</code>
<code>Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2</code>
<code>...</code>

Lock down the SSH security group to restrict access to specific IPs and change the root password.

Backup and clear

/root/.ssh/authorized_keys

.

<code>[root@VM-12-12-centos ~]# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak</code>
<code>cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied</code>

Inspect newly added users with

cat /etc/passwd

and lock suspicious accounts:

<code>[root@VM-12-12-centos ~]# usermod -L sys1</code>

Check cron jobs for hidden tasks. The attacker used various cron locations (

/var/spool/cron/

,

/etc/crontab

,

/etc/cron.d/

,

/etc/cron.*

) but some files appeared empty while logs showed recurring executions every five minutes:

<code>Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss > /dev/null 2>&1)</code>
<code>Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)</code>

Remove suspicious scripts and binaries, e.g., delete

/usr/lib/mysql/mysql

and

/sbin/httpss

, after unlocking them with

chattr -ai

because the attacker had set immutable attributes (

chattr +ia

) on many files.

<code>/tmp/chattr -ai /usr/bin/chattr</code>
<code>/tmp/chattr -ai /usr/bin/</code>

Inspect and delete a persistent malicious script located at

/.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464

that repeatedly kills security‑related processes and modifies

/etc/ld.so.preload

to load a hidden library.

<code>#!/bin/sh</code>
<code>while test 1 = 1</code>
<code>do</code>
<code>sleep 30</code>
<code>pkill -f main</code>
<code>...</code>
<code>done</code>

During dynamic linking, the loader reads the LD_PRELOAD environment variable and the /etc/ld.so.preload file, preloading any libraries listed there even if the program does not depend on them, giving the attacker a powerful persistence mechanism.

After killing the malicious process and removing the script, clear

/etc/ld.so.preload

and delete the referenced library

/usr/local/lib/libprocesshider.so

. Subsequent attempts to run commands may still show errors until the preload file is fully cleared.

3. Lessons learned

Use restrictive cloud security‑group rules; only allow necessary ports.

Enforce strong, complex passwords for SSH and management panels.

Monitor critical files (e.g., via MD5 checks) for unauthorized changes.

When a server is compromised, isolate it, restore trusted binaries from a clean system, remove immutable attributes, and audit cron jobs, SSH keys, and preload configurations.

For further reference, see cloud provider security documentation such as Tencent Cloud and Alibaba Cloud guides.