Rootkit Hijack on a Linux Server: Forensic Walkthrough and Recovery

Background

With security becoming critical in IT, this case study examines a Linux server that was compromised by a rootkit, illustrating the investigative and remediation process for operations staff.

Incident Overview

A customer’s portal server (CentOS 5.5, ports 80 and 22 open) was disconnected by the ISP after exhausting its 100 Mbps bandwidth, indicating a possible traffic‑generation attack.

Initial Findings

Network monitoring revealed outbound scans on port 80, but netstat -an showed no connections. Commands like ps -ef and top returned no suspicious processes, prompting suspicion of a rootkit.

MD5 checks of system binaries ( ps, top) against a trusted backup confirmed they had been altered, confirming a rootkit infection.

Isolation and Log Analysis

The server was taken offline. Because system commands were compromised, two approaches were considered: mounting the disk on a clean host or copying trusted binaries to a safe path. The latter was used.

Log inspection with more /var/log/secure | grep Accepted revealed a successful SSH login for the built‑in mail account from IP 62.17.163.186 (Ireland) at 03:10:25 on Oct 3, before the attack began. The /etc/shadow entry showed the mail account now had a password, indicating it had been weaponized as a hidden backdoor.

Other logs (/var/log/messages, /var/log/wtmp) were empty, suggesting the attacker had cleared them, though /var/log/secure remained.

Identifying the Malicious Process

The suspicious process .t (PID 22765, user nobody) was consuming high CPU and memory. Its executable path was discovered via:

[root@webserver ~]# /mnt/bin/ls -al /proc/22765/exe

which pointed to /var/tmp/.../apa/t. The hidden directory /var/tmp/.../ contained several rootkit files:

drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 apa</code>
<code>-rw-r--r-- 1 nobody nobody 0 Sep 29 22:09 apa.tgz</code>
<code>drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 caca</code>
<code>drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 haha</code>
<code>-rw-r--r-- 1 nobody nobody 0 Sep 29 22:10 kk.tar.gz</code>
<code>-rwxr-xr-x 1 nobody nobody 0 Sep 29 22:10 login</code>
<code>-rw-r--r-- 1 nobody nobody 0 Sep 29 22:10 login.tgz</code>
<code>-rwxr-xr-x 1 nobody nobody 0 Sep 29 22:10 z

Key components: z: a log‑cleaning tool (e.g., ./z 62.17.163.186) that erases traces of the attacker’s IP. t in apa: a backdoor that reads an ip file and scans listed addresses, turning the server into a “zombie”. haha: contains programs that replace system binaries, hiding malicious activity. login: a trojan that replaces the login program and records credentials.

Root Cause Investigation

The server ran a Java web application behind Apache 2.0.63 and Tomcat 5.5, linked via mod_jk. Apache’s access log showed requests to awstats.pl with a crafted configdir parameter:

GET /cgi-bin/awstats.pl?configdir=|echo;echo;ps+-aux%00 HTTP/1.0

and

GET /cgi-bin/awstats.pl?configdir=|echo;echo;cd+/var/tmp/.../haha;ls+-a%00 HTTP/1.0

This vulnerability allowed the attacker to execute arbitrary commands, upload the rootkit files to /var/tmp/.../, and gain persistent control.

The fix is to sanitize the configdir parameter in awstats.pl by adding a whitelist filter:

if ($QueryString =~ /configdir=([^&]+)/i) {
  $DirConfig=&DecodeEncodedString("$1");
  $DirConfig=~tr/a-z0-9_\-\/\./a-z0-9_\-\/\./cd;
}

Remediation Steps

Back up website data and perform a clean OS installation; remove unnecessary default accounts.

Switch SSH authentication to public‑key only, disabling password logins.

Upgrade Apache and install the latest stable version of Awstats.

Deploy tcp_wrappers or a host‑based firewall to restrict SSH source addresses.

Conclusion

The intrusion was caused by an Awstats script flaw, leading to a rootkit that hijacked the server, created hidden backdoors, and generated malicious outbound traffic. Thorough forensic analysis uncovered the attack chain, and the recommended hardening measures restore a trustworthy environment.