How to Detect and Recover from a Linux Server Intrusion: A Step‑by‑Step Guide

1. Server Intrusion Symptoms

A friend's website server showed sustained 100% CPU usage, high load, and services becoming unavailable, indicating a possible intrusion.

Initially I was reluctant to intervene, but after the friend offered a large payment I began the investigation.

2. Server Investigation and Handling

2.1 Possible Causes of Intrusion

Simple SSH password.

Overly permissive Tencent Cloud security group.

Simple password for the BT panel (likely not the entry point).

2.2 Investigation Steps

Use

ps -ef

/

top

to locate the process consuming the most resources. The

ps/top

commands had been replaced.

Search logs with

last

or

grep 'Accepted' /var/log/secure

to find unauthorized logins.

<code>[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
Aug 27 08:52:05 VM-12-12-centos sshd[3053]: Accepted password for root from 127.0.0.1 port 57534 ssh2
...</code>

lighthouse 腾讯云轻量服务器

The logs reveal foreign IP 34.215.138.2 successfully logged in within fewer than 500 attempts, confirming a brute‑force compromise. Remediation Measures

Restrict SSH access in the Tencent Cloud security group to specific IPs.

Change the SSH root password.

Backup and clear

/root/.ssh/authorized_keys

(the copy command failed due to permission issues).

Further steps included checking newly added users and locking them:

<code>[root@VM-12-12-centos ~]# usermod -L sys1</code>

Investigation of cron jobs showed hidden tasks in /var/log/cron executing every five minutes, but the usual cron files ( /var/spool/cron/ , /etc/crontab ) appeared empty, indicating hidden characters.

<code>Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss > /dev/null 2>&1;^M )
Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)
...</code>

Malicious scripts were found, such as /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 , which repeatedly killed security tools, removed files, and wrote a malicious library path to /etc/ld.so.preload :

<code># echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
lockr +ai /etc/ld.so.preload >/dev/null 2>&1</code>

During Linux dynamic linking, libraries listed in /etc/ld.so.preload are pre‑loaded before any program runs, giving attackers a powerful persistence mechanism.

After deleting the malicious library, the system still recreated the preload entry, suggesting a hidden cron or service was re‑injecting it. To unlock and replace locked binaries, the attacker‑locked chattr command itself was restored from a clean machine and used to remove the immutable attribute:

<code>/tmp/chattr -ai /usr/bin/chattr</code>

Subsequent commands restored functional copies of top , ps , and lsattr from a clean system.

3. Lessons Learned

When core utilities are replaced, copy clean binaries from an identical system and use them to regain control.

Hidden characters can make cron files appear empty; always view files with tools that reveal non‑printable characters.

4. Recommendations After a Server Compromise

Leverage cloud security groups to restrict access to essential ports only.

Enforce strong, complex passwords for all services, including SSH and management panels.

Monitor critical files (e.g.,

/etc/passwd

,

/etc/shadow

,

/etc/ld.so.preload

,

/root/.ssh/authorized_keys

) using checksum or integrity tools.

If compromised, immediately limit SSH to trusted IPs, examine

last

and

grep 'Accepted' /var/log/secure

for suspicious logins, verify that

ps/top

binaries are authentic, and clear any malicious entries in

/etc/ld.so.preload

(using

chattr -ia

if necessary).

Inspect startup scripts (

/etc/rc.local

, systemd services) and cron jobs for unknown entries.

Use

netstat

to identify abnormal network connections and trace back to malicious processes.