How to Detect and Recover from a Linux Server Intrusion: A Step‑by‑Step Guide

Server Intrusion Symptoms

A friend’s website server showed sustained 100% CPU usage, high load, and services becoming unavailable, indicating a possible intrusion.

Possible Causes

Weak SSH password.

Overly permissive cloud security group.

Simple password on the BaoTa control panel.

Investigation and Remediation Steps

Identify the most resource‑intensive processes

Run

ps -ef

/

top

to locate the offending services.

Find detailed intrusion traces

Check login records with

last

or

grep 'Accepted' /var/log/secure

. Example output shows foreign IP 34.215.138.2 successfully logging in multiple times.

<code>[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
...</code>

Immediate measures

Restrict SSH access in the cloud security group to specific IPs.

Change the root SSH password.

Backup and clear

/root/.ssh/authorized_keys

.

<code>[root@VM-12-12-centos ~]# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak</code>

Lock suspicious users

Lock newly created accounts, e.g.,

usermod -L sys1

.

<code>[root@VM-12-12-centos ~]# usermod -L sys1</code>

Inspect cron jobs

Check

/var/spool/cron/

,

/etc/crontab

,

/etc/cron.d/

and

/var/log/cron

. Logs revealed recurring tasks every five minutes executing hidden scripts.

<code>Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1)</code>

Remove malicious scripts and binaries

Delete or rename compromised files such as

/usr/bin/chattr

,

/usr/bin/lsattr

, and the hidden script

/.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464

that repeatedly kills security tools and modifies

/etc/ld.so.preload

.

<code># Example snippet from the malicious script
while true; do
  sleep 30
  pkill -f main
  killall main
  ...
  echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
  chattr +ai /etc/ld.so.preload
  ...
done</code>

Restore locked utilities

Copy clean versions of

chattr

,

lsattr

,

top

, and

ps

from an identical machine, then unlock them with

/tmp/chattr -ai /usr/bin/chattr

and similar commands.

<code>/tmp/chattr -ai /usr/bin/chattr</code>

Clean LD_PRELOAD

Remove the malicious entry

/usr/local/lib/libprocesshider.so

from

/etc/ld.so.preload

. After clearing, the system still regenerated the entry, indicating a hidden cron job that needed to be eliminated.

During dynamic linking, the loader reads the LD_PRELOAD environment variable and the /etc/ld.so.preload file, loading any specified libraries before others, which attackers can exploit to inject malicious code.

Key Takeaways

Use cloud security groups to restrict SSH to trusted IPs.

Enforce strong passwords for all accounts.

Monitor critical files (e.g., via MD5 checks) for unauthorized changes.

After a breach, promptly check for altered system binaries, suspicious cron jobs, and tampered

ld.so.preload

entries.

Source: https://www.cnblogs.com/operationhome/p/16637763.html