How I Stopped a Massive Linux DDoS Attack and Eradicated Hidden Rootkits

Incident Background

Operating system: Ubuntu 12.04 x64. Business: company system, crawler program, data queue. The server was hosted in a remote data center. Suddenly, frequent ping‑monitoring alerts indicated unreachable servers and traffic surged to about 800 MB, making SSH login impossible.

1. Problem Investigation

Initially considered cutting external network, but that would hide the attack source. Contacted the data‑center staff to log in. Checked the w command for abnormal users, examined /var/log/auth.log (already cleared), and used iftop to identify high‑traffic connections.

The analysis showed continuous HTTP requests to IP 104.31.225.6.

Blocked the IP with iptables: iptables -A OUTPUT -d 104.31.225.6 -j DROP Traffic dropped temporarily, then resumed with a different destination IP.

2. Locating the Attack Source

Used netstat -atup | grep 15773 but found no result; the connections were short‑lived. Switched to lsof -i :15773 to obtain the PID, then ps -ef to list related files.

Found no suspicious processes. Suspected a rootkit, so compared md5 checksums of system binaries. The size of /bin/lsof was 1.2 M, indicating tampering.

Replaced compromised tools (netstat, ps, etc.) with clean versions from a trusted system.

3. Removing the Malware

After replacing the tools, ps -ef revealed suspicious entries such as:

/sbin/java.log

/usr/bin/dpkgd/ps

/usr/bin/bsd-port/getty

/usr/sbin/.sshd

Killed these processes and deleted their files and directories.

Discovered that /etc/init.d contained an abnormal script that relaunched the java.log backdoor; removed the script and the file.

4. Incident Summary

Replaced compromised utilities (netstat, lsof, ps, ss). Identified malicious files:

/sbin/java.log – packet‑sending program that regenerated.

/usr/bin/dpkgd – replaced utility.

/usr/bin/bsd-port – generated java.log or backdoor.

/usr/sbin/.sshd – backdoor.

After cleanup, external traffic returned to normal.

Prevention Recommendations

Enable firewall and allow only trusted sources.

Collect system, login, and application logs.

Monitor user logins, password retries, and command execution.

Watch critical files (/etc/passwd, /etc/shadow, /web, /tmp) for changes.

Monitor process creation and alert on suspicious processes.

Regularly scan servers and web applications for vulnerabilities.

Security is an ongoing effort; reduce the attack surface and strengthen system protection.