How to Detect and Remediate a Linux Rootkit Attack: A Step‑by‑Step Case Study

1. Attack Symptoms

A customer’s portal server in a telecom data center suddenly lost network connectivity because the ISP cut the line after the server exhausted the 100 Mbps bandwidth by sending massive outbound traffic. The server runs CentOS 5.5 with ports 80 and 22 open, suggesting a possible traffic‑based attack.

2. Preliminary Analysis

Network monitoring confirmed outbound scans on port 80, yet

netstat –an

showed no connections to that port. Further checks with

ps –ef

and

top

revealed no suspicious processes. MD5 checksums of system binaries (ps, top) compared to a clean backup showed they had been altered, indicating a rootkit infection.

3. Isolating the System

The compromised server was disconnected from the network. Because system commands were replaced, two mitigation methods were considered: (1) mount the server’s disk on a clean host for analysis, or (2) copy trusted binaries from a clean system to a separate directory and invoke them with full paths. The second method was used.

Login logs were examined with:

more /var/log/secure | grep Accepted

The log showed a successful SSH login at 03:10:25 on Oct 3 from IP 62.17.163.186 using the built‑in “mail” account, which normally cannot log in. The /etc/shadow entry confirmed the mail account had been given a password, suggesting the attacker created a hidden backdoor account.

4. Tracing the Attack Source

Running

ps

revealed a suspicious process:

nobody   22765 1 6 Sep29 ? 4-00:11:58 .t

The process

.t

(PID 22765) consumed high CPU and memory. Examining

/proc/22765/exe

pointed to an executable hidden in

/var/tmp/…/apa/t

. Listing the hidden directory showed several rootkit files (e.g.,

z

,

t

,

login

, etc.).

z : clears logs for a given IP.

t : backdoor that reads an

ip

file and scans listed hosts, turning the server into a bot.

haha directory: contains binaries that replace system commands, hiding malicious activity.

login : a trojan that replaces the login program and records credentials.

5. Identifying the Initial Compromise

The server runs a Java web application behind Apache 2.0.63 and Tomcat 5.5, integrated via

mod_jk

. Apache serves static content on port 80. Inspection of

access.log

revealed malicious requests to

awstats.pl

:

62.17.163.186 - - [29/Sep/2013:22:17:06 +0800] "GET /cgi-bin/awstats.pl?configdir=|echo;echo;ps+-aux%00 HTTP/1.0" 200 12333 "-" "Mozilla/5.0 ..."
62.17.163.186 - - [29/Sep/2013:22:17:35 +0800] "GET /cgi-bin/awstats.pl?configdir=|echo;echo;cd+/var/tmp/.../haha;ls+-a%00 HTTP/1.0" 200 1626 "-" "Mozilla/5.0 ..."

The

configdir

parameter in

awstats.pl

allowed command injection, enabling the attacker to execute arbitrary shell commands, create hidden directories, and upload the rootkit files.

The fix involves sanitizing the parameter, e.g.:

if ($QueryString =~ /configdir=([^&]+)/i) {
  $DirConfig=&DecodeEncodedString("$1");
  $DirConfig=~tr/a-z0-9_\-\/\./a-z0-9_\-\/\./cd;
}

6. Summary of the Attack Chain

1) Exploited the Awstats

configdir

command‑injection vulnerability to gain a shell. 2) Created a hidden directory under

/var/tmp

and uploaded rootkit components. 3) Gained root privileges via backdoor binaries, using the server to send outbound traffic. 4) Modified the default

mail

account to allow remote login. 5) Cleaned logs with the

z

program, leaving limited forensic evidence.

7. Recovery Recommendations

Back up website data and perform a clean OS reinstall.

Remove unnecessary default accounts and enforce key‑based SSH authentication.

Upgrade Apache and Awstats to the latest stable versions.

Deploy TCP Wrappers or a host‑based firewall to restrict SSH source addresses.