How to Analyze and Recover from a Linux Rootkit Intrusion

Case Study: Linux Server Compromised by Rootkit

With information security becoming critical, especially after incidents like the "Prism" leaks, operations staff must adopt attacker‑mindset guidelines to patch threats. This case demonstrates the handling of a server infected with a rootkit, a common Linux attack vector.

1. Symptoms of Attack

A client’s portal server in a telecom data center was cut off because it exhausted a 100 M bandwidth by continuously sending outbound packets. The server (CentOS 5.5) had ports 80 and 22 open, yet normal traffic could not explain the bandwidth drain, suggesting a traffic‑based attack.

2. Preliminary Analysis

Network monitoring revealed outbound scans on port 80, but netstat -an showed no connections on that port. Commands like ps -ef and top returned no suspicious processes. MD5 checksums of these binaries compared with a clean system showed they had been altered, confirming a rootkit installation.

3. Isolating the System

Because the server kept sending packets, the first step was to disconnect it from the network and examine logs. Since system binaries were compromised, two approaches were possible: mount the disk on a clean host or copy trusted binaries to a safe path and invoke them with full paths. The latter method was used. more /var/log/secure |grep Accepted The log revealed a successful SSH login at 03:10:25 on October 3 from IP 62.17.163.186 using the built‑in mail account, which normally cannot log in. This indicated prior unauthorized access.

Oct 3 03:10:25 webserver sshd[20701]: Accepted password for mail from 62.17.163.186 port 53349 ssh2

The /etc/shadow file showed the mail account now had a password, confirming it was weaponized as a hidden backdoor.

mail:$1$kCEd3yD6$W1evaY5BMPQIqfTwTVJiX1:15400:0:99999:7:::

Further log inspection showed cleared /var/log/messages and /var/log/wtmp, while /var/log/secure remained untouched.

4. Tracing the Attack Source

Running ps and top uncovered a suspicious process named .t owned by nobody, consuming high CPU and memory for several days.

PID USER   PR  NI  VIRT   RES   SHR S %CPU %MEM   TIME+ COMMAND
22765 nobody 15   0 1740m 1362m 1228 S 98.3 91.5 2892:19 .t

Inspecting /proc/22765/exe revealed the executable path /var/tmp/…/apa/t, hidden within a directory under /var/tmp. Listing that directory showed multiple malicious files:

drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 apa
-rw-r--r-- 1 nobody nobody     0 Sep 29 22:09 apa.tgz
drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 caca
drwxr-xr-x 2 nobody nobody 4096 Sep 29 22:09 haha
-rw-r--r-- 1 nobody nobody     0 Sep 29 22:10 kk.tar.gz
-rwxr-xr-x 1 nobody nobody     0 Sep 29 22:10 login
-rw-r--r-- 1 nobody nobody     0 Sep 29 22:10 login.tgz
-rwxr-xr-x 1 nobody nobody     0 Sep 29 22:10 z

Analysis identified the roles of these files:

The z program deletes logs related to the attacker’s IP.

The t program in apa reads an ip file and scans listed IPs, turning the server into a bot.

The haha directory contains binaries that replace system commands, hiding malicious activity.

The login binary is a trojan that records login credentials.

5. Identifying the Initial Exploit

The server ran a Java web application behind Apache 2.0.63, Tomcat 5.5, and the mod_jk connector. Apache handled static requests, so the breach likely originated from Apache. Examining access.log uncovered malicious requests exploiting a vulnerability in the awstats.pl script’s configdir parameter:

62.17.163.186 - - [29/Sep/2013:22:17:06 +0800] "GET http://www.xxx.com/cgi-bin/awstats.pl?configdir=|echo;echo;ps+-aux%00 HTTP/1.0" 200 12333 "-" "Mozilla/5.0 ..."
62.17.163.186 - - [29/Sep/213:22:17:35 +0800] "GET http://www.xxx.com/cgi-bin/awstats.pl?configdir=|echo;echo;cd+/var/tmp/.../haha;ls+-a%00 HTTP/1.0" 200 1626 "-" "Mozilla/5.0 ..."

The vulnerable awstats.pl allowed command injection, enabling the attacker to create hidden directories under /var/tmp and upload the rootkit files.

6. Patching the Vulnerability

To fix the issue, modify awstats.pl as follows:

if ($QueryString =~ /configdir=([^&]+)/i)
{
  $DirConfig=&DecodeEncodedString("$1");
  $DirConfig=~tr/a-z0-9_-/./a-z0-9_-/./cd;
}

7. How to Recover the Website

Because the system files have been altered, the safest approach is to rebuild:

Install a stable OS version and remove unnecessary default accounts.

Switch SSH authentication to public‑key only.

Upgrade Apache and install the latest stable Awstats package.

Deploy tcp_wrappers or a comparable firewall to restrict SSH source addresses.