How a Redis Hijack Exposed Critical Security Gaps and What to Do

Problem Overview

During a deployment, several Redis instances on development, testing, and production servers failed to start, severely impacting development and online services. Initial checks using ps -ef | grep redis and Redis Desktop Manager revealed that the services were down.

Investigation Process

Identified that all Redis instances using the default port 6379 were down, with server 246 being the most affected.

Checked server performance with top and discovered a malicious process consuming high CPU.

Found suspicious cron jobs via crontab -r on server 246.

Deleted the root cron file in /var/spool/cron and killed the malicious process (named AnXqV.yam ).

Observed the process respawn after a few minutes, indicating persistence.

Located numerous files named AnXqV across the filesystem, confirming a mining infection.

Downloaded a script ( pm.sh) from the attacker’s URL and examined its contents.

Removed malicious files including /var/spool/cron/root, /var/spool/cron/crontabs/root, ~/.ssh/authorized_keys, /var/spool/cron/authorized_keys, and several files in /tmp.

Analyzed log files showing attempts to access malicious sites, DNS queries, and IP lookups pointing to a DigitalOcean server used by the attacker.

Checked other Redis servers for similar artifacts and found a suspicious cron file on server 204.

Deleted the malicious files and Redis dump files ( /tmp/dump.rdb, /usr/local/redis/bin/dump.rdb, /var/spool/cron/dump.rdb) and restored Redis services.

Redis Protection Recommendations

Do not use the default port 6379.

Enable Redis authentication with username and password.

Bind Redis to specific IP addresses in the configuration file.

Keep Redis servers off the public internet.

Enable firewall rules (e.g., iptables) to restrict access.

Consider containerizing Redis (e.g., Docker) to simplify recovery after compromise.

References

CentOS7 installation of Redis compromised by AnXqV mining program.

Solutions for Alibaba Cloud servers infected by minerd.

Overview of Google infrastructure security design.

Dockerized alternatives to Redis such as Codis and Pika.

Conclusion

Security is paramount; thorough investigation and swift remediation are essential. Prioritize stabilizing the development environment while addressing the root cause, and implement the recommended hardening measures to mitigate future risks.