Server Compromise Case Study: Forensic Analysis and Recovery from Rootkit Persistence

Incident Overview

A friend's self-hosted website server exhibited sustained 100% CPU usage and service unavailability. Initial investigation revealed the ps and top commands had been replaced, preventing normal process inspection.

Root Cause Analysis

Entry Vector

SSH brute-force success from IP 34.215.138.2 (AWS range) after fewer than 500 attempts, enabled by weak root password and overly permissive Tencent Cloud security group (SSH open to all IPs). Baota panel also used a simple password, though likely not the entry point.

Evidence from /var/log/secure

Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2

Additional logins from localhost (127.0.0.1) and Tencent Cloud Lighthouse IPs appeared afterward.

Investigation Steps

1. User Account Enumeration

cat /etc/passwd

revealed unauthorized users (e.g., sys1). Locked via usermod -L sys1.

2. Cron Job Analysis

Standard locations ( /var/spool/cron/, /etc/crontab, /etc/cron.d/) appeared empty, yet /var/log/cron showed execution every 5 minutes:

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M)

Hidden cron files used special characters and were misidentified as data type by file command, evading cat / more.

3. Startup Script Persistence

/etc/rc.local

contained /usr/bin/0f4f80f9ab start (file missing). Commented out.

4. Binary Replacement & File Locking

Attacker replaced top, ps, chattr, lsattr and applied immutable/append-only attributes ( chattr +ia) on both files and parent directory /usr/bin/.

5. Recovery of System Binaries

Copied clean chattr, lsattr from identical OS version to

/tmp

chmod +x /tmp/chattr

Unlocked directory first: /tmp/chattr -ai /usr/bin/ Then unlocked and replaced /usr/bin/chattr: /tmp/chattr -ai /usr/bin/chattr Restored top, ps, lsattr similarly

6. LD_PRELOAD Rootkit

Malicious script ( /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464) ran infinite loop every 30 seconds, killing competitor miners (xmrig, cpuminer, etc.) and enforcing persistence via:

echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload

Per cited reference, /etc/ld.so.preload forces preloading of specified shared libraries before any other, enabling hooking of system calls to hide processes/files.

Deleting /usr/local/lib/libprocesshider.so caused command errors until /etc/ld.so.preload was cleared. A surviving process kept rewriting it; killing that process stopped recurrence.

Key Lessons & Recommendations

Preventive Measures

Restrict SSH via security groups to known IPs only

Enforce complex passwords / key-based auth

Monitor critical file integrity (md5) for: /etc/passwd, /etc/shadow,

/etc/group

/root/.ssh/authorized_keys

,

/etc/ssh/sshd_config

/etc/crontab

, /var/spool/cron/root,

/etc/cron.d/

/etc/ld.so.preload

, /etc/rc.local Core binaries: ps, top, ls, netstat, lsof, chattr, lsattr, last, history, sudo,

pstree

Incident Response Checklist

Restrict SSH access immediately (security group / firewall)

Check last and grep 'Accepted' /var/log/secure for unauthorized logins

Lock suspicious users ( usermod -L)

Verify ps / top integrity; replace from clean host if tampered

Inspect /etc/ld.so.preload; clear and delete referenced .so

If files immutable, use chattr -ia (copy chattr from clean host if needed)

Check cron locations + /var/log/cron for hidden jobs

Review /etc/rc.local and systemd services

Use netstat / ss for anomalous connections

Kill malicious processes, delete scripts, remove persistence

References: Tencent Cloud security hardening guide (https://cloud.tencent.com/document/product/296/9604), Alibaba Cloud incident response doc (https://help.aliyun.com/document_detail/40994.htm).