Debug Kubernetes Pods Without Restart Using Ephemeral Containers

Debugging a pod by kubectl exec is simple but has drawbacks: the running container may lack necessary tools, extra permissions require a pod restart, and adding tools to the image introduces security risks.

Using Ephemeral Containers

When kubectl exec is insufficient, an ephemeral container can be added to a running pod without restarting it. The new container shares the pod’s network and process namespaces, volumes, and node.

Setup a Kind Cluster

Create a Kubernetes 1.23 cluster with Kind:

kind create cluster

Verify the cluster is running and access the node:

docker exec -it <kind-container-id> bash

Create a Simple Workload

Deploy a single‑replica Nginx deployment:

kubectl create deployment nginx --image=nginx

Network Troubleshooting

Attach an ephemeral container using the nicolaka/netshoot image, which includes tools like tcpdump and strace:

kubectl debug -it pod-name --image=nicolaka/netshoot -- tcpdump -n port 80

Capture packets from the Nginx container and observe them in the temporary container’s output.

Process Tracing

To trace processes, the temporary container must share the pod’s process namespace and have the SYS_PTRACE capability. Use:

kubectl debug -it <pod-name> --image=nicolaka/netshoot --target <container-name> -- bash

Because the container lacks ptrace permission, a JSON PATCH request is sent to the API server to add SYS_PTRACE to the ephemeral container’s securityContext:

curl -v -XPATCH -H "Content-Type: application/json-patch+json" \
  http://127.0.0.1:8001/api/v1/namespaces/default/pods/nginx-.../ephemeralcontainers \
  --data-binary @- <<EOF
[{"op":"add","path":"/spec/ephemeralContainers/-","value":{"command":["/bin/sh"],"stdin":true,"tty":true,"image":"nicolaka/netshoot","name":"debug-strace","securityContext":{"capabilities":{"add":["SYS_PTRACE"]}},"targetContainerName":"nginx"}}]
EOF

After adding the capability, strace can trace Nginx processes.

Node Debugging

An ephemeral container can also be attached to a node for host‑level debugging:

kubectl debug node/<node-name> -it --image=<image-name>

The container runs in the host’s IPC, network, and PID namespaces, with the node’s root filesystem mounted at /host. Use chroot /host to operate directly on the node.

These techniques enable comprehensive debugging of Kubernetes workloads without restarting pods.