How to Investigate and Respond to Kubernetes Cluster Intrusions

1. Overview

Kubernetes clusters are a frequent target for attackers. When a breach is detected, rapid incident response at the pod, node, and cluster levels is required to contain the impact and restore service.

2. Pod‑level investigation

2.1 Access the compromised container

The quickest way to run commands inside a pod is: kubectl exec -it <pod-name> -- bash This method works only if the container already includes the needed debugging tools.

2.2 Debug with a temporary container

Kubernetes 1.20+ provides kubectl debug to launch a sidecar container that shares the pod’s network and PID namespace. The sidecar can mount the target container’s root filesystem at /proc/1/root, giving full access without disturbing the running workload.

kubectl debug -it <pod-name> \
  --image=busybox:latest \
  --target=<primary-container-name>

2.3 Retrieve pod logs

Even if the container is not reachable, logs are always available through the API server:

kubectl logs <pod-name> [<container-name>] --previous

Use the --previous flag to view logs from a crashed container.

3. Node (host) investigation

3.1 Node debugging without SSH

kubectl debug node

creates a privileged pod on the target node. The pod’s root filesystem is mounted at /host, allowing direct inspection of host binaries, configuration files, and running processes.

kubectl debug -it node/<node-name> \
  --image=busybox:latest \
  --privileged

3.2 Relevant host log locations

Container stdout/stderr: /var/lib/docker/containers/<container-id>/-json.log Aggregated container logs: /var/log/containers/*.log Kubelet logs:

journalctl -u kubelet

4. Cluster‑wide audit log analysis

Kubernetes audit logging records every API‑server request. When investigating a breach, filter on fields such as sourceIPs, user, userAgent, and verb to spot malicious activity.

4.1 Unauthorized API‑server access

{"user":{"username":"system:unsecured","groups":["system:masters","system:authenticated"]}}

4.2 Use of a leaked service‑account token

"user.username"="system:serviceaccount:test:bypass"
userAgent="curl/7.68.0"

4.3 Use of a leaked kubeconfig

"sourceIPs"="192.168.44.133"
userAgent="kubectl/v1.28.2 (linux/amd64) kubernetes/89a4ea3"

4.4 Creation of privileged pods

verb=create responseStatus.code=201
requestObject.spec.containers[].securityContext.privileged=true

4.5 HostPath volume mounting

verb=create responseStatus.code=201
responseObject.spec.volumes[].hostPath.path="/"

4.6 Creation of CronJobs

verb=create responseStatus.code=201
objectRef.resource=cronjobs

These patterns help reconstruct the attacker’s actions, identify persistence mechanisms, and assess the scope of compromise.

5. Summary

Effective Kubernetes incident response combines direct pod inspection (via kubectl exec or kubectl debug), host‑level debugging with kubectl debug node, and systematic audit‑log queries. By correlating container state, host artifacts, and API‑server events, responders can locate footholds, trace the attack path, and safely restore cluster operations.