Server Intrusion Investigation and Remediation on a Compromised Linux Host

The author describes a friend’s Linux server that was likely compromised, showing symptoms such as sustained 100% CPU usage and inability of services to respond.

Initial investigation identified weak SSH passwords, overly permissive cloud security group rules, and a simple password on the Baota control panel as possible entry points.

Investigation steps included using ps -ef and top to locate high‑resource processes, examining /var/log/secure with grep 'Accepted' to find unauthorized logins from foreign IPs, and checking last for login history.

Key remediation measures were applied:

Restricted SSH access in the cloud security group to specific IPs.

Changed the root SSH password.

Backed up and cleared /root/.ssh/authorized_keys .

Further actions involved locking suspicious users with usermod -L sys1 , inspecting newly added users via cat /etc/passwd , and analyzing cron jobs. The author noted that cron configuration files appeared empty, yet logs showed recurring tasks every five minutes.

Malicious scripts were discovered, including a persistent loop that killed legitimate processes and repeatedly wrote a malicious library path to /etc/ld.so.preload . The script also attempted to delete security tools and stop related services.

To restore system utilities that had been replaced or locked (e.g., ps , top , chattr , lsattr ), the author copied clean binaries from an identical machine to /tmp , used chattr -ai to unlock them, and then replaced the compromised files.

Additional hardening recommendations were provided:

Use minimal security‑group rules for critical ports.

Enforce strong passwords for all accounts.

Monitor critical files (e.g., /etc/passwd, /etc/shadow, /etc/ld.so.preload) for changes.

Regularly check processes, network connections, and scheduled tasks.

The article concludes with links to cloud provider security documentation and a reminder to use chattr -ia when files become immutable.