How I Rescued a Compromised Ubuntu Server: Step‑by‑Step Intrusion Response

Background

Operating System: Ubuntu 12.04 x64

Running Services: Company business system, crawler program, data queue.

The server was hosted in an external data center. Suddenly, monitoring emails reported ping failures and traffic spiked to about 800 MB, making SSH login impossible.

1. Problem Investigation

Initially tried to cut the external network, but that would hide the traffic source. Contacted the data‑center staff and allowed them to log in. Checked /var/log/auth.log (already cleared) and used iftop to identify traffic. Found continuous HTTP traffic to IP 104.31.225.6. iptables -A OUTPUT -d 104.31.225.6 -j DROP Blocking the IP temporarily reduced traffic, but it later resumed from a different IP.

2. Locating the Attack Source

Used netstat to filter ports, but short‑lived connections were not visible. Employed lsof -i :15773 to find the PID, then lsof -p PID to inspect opened files. netstat -atup | grep 15773 Compared file sizes with a clean system and discovered abnormal binaries (e.g., /bin/lsof size 1.2 MB). Downloaded clean versions of netstat, ps, etc., and replaced the compromised tools.

3. Removing the Malware

After running ps -ef, several suspicious processes appeared:

pid /sbin/java.log pid /usr/bin/dpkgd/ps -ef pid /usr/bin/bsd-port/getty pid /usr/bin/.sshd

Killed and deleted these files. Noted that java.log reappeared, indicating a persistence script in /etc/init.d. Removed the script and the file, stopping the backdoor.

Incident Summary

ls /usr/bin/dpkgd/ – replaced tool, not present in a clean system. Compromised binaries: netstat, lsof, ps, ss. /sbin/java.log – packet‑sending program, regenerates after deletion. /usr/bin/bsd-port – generates java.log or other backdoor. /usr/sbin/.sshd – backdoor program.

Preventive Measures

Enable firewall after installation, allow only trusted sources, remove unnecessary users and services.

Collect logs (system, authentication, application) for early risk detection.

Monitor user logins, password retries, and command execution.

Watch critical files/directories (e.g., /etc/passwd, /etc/shadow, /web, /tmp) for changes.

Monitor process states and alert on new or suspicious processes.

Regularly scan servers and web applications for vulnerabilities.

There is no absolute security; the goal is to reduce the attack surface and improve system protection.