Debug Running Kubernetes Pods with Ephemeral Containers: A Step‑by‑Step Guide

Debugging a running container or pod is not as straightforward as debugging a regular process; this guide shows how to use temporary (ephemeral) containers that share the target pod's namespaces to debug business container processes.

Directly exec‑ing into a problematic pod is simple but has drawbacks: the pod may lack debugging tools, additional permissions require a pod restart, and adding tools to the image or raising container privileges introduces security risks.

Using Ephemeral Debug Containers

When

kubectl exec

is insufficient—e.g., the container has crashed, uses a distroless image, or lacks required privileges—ephemeral containers provide an interactive troubleshooting method without restarting the pod.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

Ephemeral containers are added to an existing pod with a custom image and share the target container’s resources, including Linux network and process namespaces, shared volumes, and node access.

Creating a Kind Cluster

First, set up a Kubernetes 1.23 cluster using

kind create cluster

. Verify the cluster is running and accessible.

All operations are performed from the main kind node via

docker exec -it <node> bash

.

Creating a Simple Workload

Deploy a single‑replica Nginx deployment:

<code>kubectl create deployment nginx --image=nginx</code>

Diagnosing Network Issues

Network debugging requires sharing the network namespace, which is automatically shared when an ephemeral container is attached.

Create the first ephemeral container using the

nicolaka/netshoot

image, which includes tools like

tcpdump

and

strace

:

<code>kubectl debug -it pod-name --image=nicolaka/netshoot --command -- /bin/sh</code>

Verify both containers share the same Linux namespace:

<code>systemd-cgls -u kubelet-kubepods-besteffort.slice</code>

Capture network packets from the Nginx container:

<code>tcpdump -n port 80</code>

Send a request to the pod and observe the packets printed in the ephemeral container’s terminal.

<code>curl http://pod-ip-address</code>

Tracing Processes with Ephemeral Containers

To trace processes, both containers must share the same process namespace and the ephemeral container needs the

SYS_PTRACE

capability.

Create the container with the

--target

flag:

<code>kubectl debug -it &lt;pod-name&gt; --image=nicolaka/netshoot --target &lt;container-name&gt; -- bash</code>

Because the container lacks

SYS_PTRACE

, we add the capability via a PATCH request to the Kubernetes API:

<code>curl -v -XPATCH -H "Content-Type: application/json-patch+json" \
  'http://127.0.0.1:8001/api/v1/namespaces/default/pods/nginx-8f458dc5b-wkvq4/ephemeralcontainers' \
  --data-binary @- <<EOF
[{
  "op": "add", "path": "/spec/ephemeralContainers/-",
  "value": {
    "command":[ "/bin/sh" ],
    "stdin": true, "tty": true,
    "image": "nicolaka/netshoot",
    "name": "debug-strace",
    "securityContext": {"capabilities": {"add": ["SYS_PTRACE"]}},
    "targetContainerName": "nginx"
  }
}]
EOF</code>

Now

strace

can be run inside the ephemeral container to trace Nginx processes.

Debugging Nodes via Ephemeral Containers

If SSH access to a node is unavailable, an ephemeral container can be used to debug the node:

<code>kubectl debug node/&lt;node-name&gt; -it --image=&lt;image-name&gt;</code>

The container runs in the host’s IPC, network, and PID namespaces, and the node’s root filesystem is mounted at

/host

. To make the container’s root filesystem identical to the node’s, execute

chroot /host

inside the container.

Source: https://juejin.cn/post/7151657396967276580