How a Linux Server Became a Botnet: A Step‑by‑Step Rootkit Forensics Walkthrough

Incident Overview

A customer’s portal server hosted in a telecom data center suddenly exhausted its 100 Mbps bandwidth, prompting the ISP to cut the connection. The CentOS 5.5 machine exposed ports 80 and 22, yet normal traffic could not explain the bandwidth drain, indicating a possible traffic‑based attack.

1. Attack Symptoms

Network monitoring revealed outbound scans from port 80, but netstat -an showed no active connections on that port. Commands such as ps -ef and top returned no suspicious processes, raising suspicion of a rootkit.

2. Preliminary Analysis

MD5 checksums of system binaries (e.g., ps, top) differed from a trusted backup, confirming that the binaries had been replaced and that a rootkit was present.

3. Isolating the System

The server was disconnected from the network. Because the compromised binaries could not be trusted, two approaches were considered: (1) mount the disk on a clean host for offline analysis, or (2) copy trusted binaries from a clean system to a separate directory and invoke them with absolute paths. The second method was used.

4. Tracing the Attack Source

Log inspection ( more /var/log/secure | grep Accepted) revealed a successful SSH login at 03:10:25 on Oct 3 from IP 62.17.163.186 using the built‑in mail account, which should not permit remote login. The /etc/shadow entry showed the mail password had been set, indicating the attacker created a hidden backdoor account.

Further log checks showed /var/log/messages and /var/log/wtmp were empty, suggesting the attacker had cleared them, while /var/log/secure remained untouched.

5. Identifying Malicious Processes

The process list displayed a suspicious .t program running under the nobody user:

PID USER   PR NI VIRT   RES   SHR S %CPU %MEM   TIME+   COMMAND
22765 nobody 15 0 1740m 1362m 1228 S 98.3 91.5 2892:19 .t

Inspecting /proc/22765/exe revealed the executable path /var/tmp/.../apa/t. The hidden directory under /var/tmp contained multiple rootkit files:

drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 apa
-rw-r--r-- 1 nobody nobody   0 Sep 29 22:09 apa.tgz
...

Key components: z: a log‑clearing utility ( ./z 62.17.163.186) that erases all traces of the attacker’s IP. t (in apa): a backdoor that reads an ip file and scans listed IPs, turning the server into a “zombie”. haha directory: contains programs that replace system binaries, hiding malicious activity. login: a trojanized login binary that records credentials.

6. Root Cause Investigation

The server ran a Java‑based web stack: Apache 2.0.63, Tomcat 5.5, linked via mod_jk. Apache exposed port 80, while Tomcat did not. Examination of access.log uncovered malicious requests to awstats.pl:

62.17.163.186 - - [29/Sep/2013:22:17:06 +0800] "GET /cgi-bin/awstats.pl?configdir=|echo;echo;ps+-aux%00" ...
62.17.163.186 - - [29/Sep/2013:22:17:35 +0800] "GET /cgi-bin/awstats.pl?configdir=|echo;echo;cd+/var/tmp/.../haha;ls+-a%00" ...

The configdir parameter in awstats.pl was vulnerable, allowing command injection. The attacker used this flaw to upload the rootkit files into /var/tmp/.../.

Patch: modify the Perl script to sanitize configdir:

if ($QueryString =~ /configdir=([^&]+)/i) {
    $DirConfig=&DecodeEncodedString("$1");
    $DirConfig=~tr/a-z0-9_\-\/\./a-z0-9_\-\/\./cd;
}

7. Remediation Steps

Because the system integrity was compromised, the recommended recovery procedure is:

Reinstall a clean, supported operating system and remove unnecessary default accounts.

Switch SSH authentication to public‑key only, disabling password logins.

Upgrade Apache and install the latest stable version of Awstats.

Deploy tcpd (TCP Wrappers) or a host‑based firewall to restrict SSH source addresses.

After rebuilding, ensure all binaries are verified, logs are monitored, and the Awstats vulnerability is patched to prevent recurrence.