Debug Running Kubernetes Pods with Ephemeral Containers: A Step‑by‑Step Guide

Debugging a running container or pod is not as straightforward as debugging a regular process; this guide shows how to use temporary (ephemeral) containers that share the target pod's namespaces to debug business container processes.

Directly exec‑ing into a problematic pod is simple but has drawbacks: the pod may lack debugging tools, additional permissions require a pod restart, and adding tools to the image or raising container privileges introduces security risks.

Using Ephemeral Debug Containers

When kubectl exec is insufficient—e.g., the container has crashed, uses a distroless image, or lacks required privileges—ephemeral containers provide an interactive troubleshooting method without restarting the pod.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

Ephemeral containers are added to an existing pod with a custom image and share the target container’s resources, including Linux network and process namespaces, shared volumes, and node access.

Creating a Kind Cluster

First, set up a Kubernetes 1.23 cluster using kind create cluster. Verify the cluster is running and accessible.

All operations are performed from the main kind node via docker exec -it <node> bash.

Creating a Simple Workload

Deploy a single‑replica Nginx deployment:

kubectl create deployment nginx --image=nginx

Diagnosing Network Issues

Network debugging requires sharing the network namespace, which is automatically shared when an ephemeral container is attached.

Create the first ephemeral container using the nicolaka/netshoot image, which includes tools like tcpdump and strace:

kubectl debug -it pod-name --image=nicolaka/netshoot --command -- /bin/sh

Verify both containers share the same Linux namespace:

systemd-cgls -u kubelet-kubepods-besteffort.slice

Capture network packets from the Nginx container: tcpdump -n port 80 Send a request to the pod and observe the packets printed in the ephemeral container’s terminal.

curl http://pod-ip-address

Tracing Processes with Ephemeral Containers

To trace processes, both containers must share the same process namespace and the ephemeral container needs the SYS_PTRACE capability.

Create the container with the --target flag:

kubectl debug -it <pod-name> --image=nicolaka/netshoot --target <container-name> -- bash

Because the container lacks SYS_PTRACE, we add the capability via a PATCH request to the Kubernetes API:

curl -v -XPATCH -H "Content-Type: application/json-patch+json" \
  'http://127.0.0.1:8001/api/v1/namespaces/default/pods/nginx-8f458dc5b-wkvq4/ephemeralcontainers' \
  --data-binary @- <<EOF
[{
  "op": "add", "path": "/spec/ephemeralContainers/-",
  "value": {
    "command":[ "/bin/sh" ],
    "stdin": true, "tty": true,
    "image": "nicolaka/netshoot",
    "name": "debug-strace",
    "securityContext": {"capabilities": {"add": ["SYS_PTRACE"]}},
    "targetContainerName": "nginx"
  }
}]
EOF

Now strace can be run inside the ephemeral container to trace Nginx processes.

Debugging Nodes via Ephemeral Containers

If SSH access to a node is unavailable, an ephemeral container can be used to debug the node:

kubectl debug node/<node-name> -it --image=<image-name>

The container runs in the host’s IPC, network, and PID namespaces, and the node’s root filesystem is mounted at /host. To make the container’s root filesystem identical to the node’s, execute chroot /host inside the container.

Source: https://juejin.cn/post/7151657396967276580