Master Kubernetes Debugging with Ephemeral Containers: A Step‑by‑Step Guide

Container images are often stripped down to reduce attack surface, improve scan speed, shrink size, accelerate builds, and minimize dependencies, which leaves them without debugging tools. Kubernetes Ephemeral Containers solve this by allowing a temporary container that includes any needed tools to be added to a running Pod for troubleshooting.

Ephemeral Container Configuration

Ephemeral containers share the same pod spec as regular containers, but several fields are disabled and behaviors changed. Notable differences include:

They do not restart.

Resource limits cannot be defined.

Ports cannot be used.

Startup, liveness, and readiness probes are disallowed.

Starting an Ephemeral Container

First, verify that the Ephemeral Containers feature is enabled. kubectl debug -it <POD_NAME> --image=busybox If the feature is disabled, you will see an error like:

Defaulting debug container name to debugger-wg54p.
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").

Enable the feature by adding EphemeralContainers=true to the --feature-gates flag of the kubelet, kube‑apiserver, kube‑controller‑manager, kube‑proxy, and kube‑scheduler, e.g.:

... 
--feature-gates=RemoveSelfLink=false,EphemeralContainers=true 
...

Using an Ephemeral Container

Once the cluster supports Ephemeral Containers, you can create one. Example workflow:

kubectl create deployment nginx-deployment --image=nginx

kubectl get pods

Assuming the pod name is nginx-deployment-66b6c48dd5-frsv9, launch a temporary busybox container attached to it:

kubectl debug -it pods/nginx-deployment-66b6c48dd5-frsv9 --image=busybox

Inside the debug container you can run commands such as:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=112 time=9.8 ms
...
# nc --help
Usage: nc [OPTIONS] HOST PORT  - connect
...

Running kubectl describe pod <POD_NAME> now shows an "Ephemeral Containers" section with details of the debug container.

Ephemeral Containers:
  debugger-thwrn:
    Container ID:   containerd://eec23aa9ee63d96b82970bb947b29cbacc30685bbc3418ba840dee109f871bf0
    Image:          busybox
    Image ID:       docker.io/library/busybox@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353
    Port:           <none>
    Host Port:      <none>

Sharing the Process Namespace with an Ephemeral Container

Process‑namespace sharing enhances debugging but cannot be applied to existing containers directly. Create a copy of the target pod with --share-processes and --copy-to:

kubectl debug -it <POD_NAME> --image=busybox --share-processes --copy-to=debug-pod

Running ps aux inside the debug container shows processes from both the original and the pause container:

# ps aux

PID   USER   TIME  COMMAND
  1   root    0:00 /pause
  6   root    0:00 nginx: master process nginx -g daemon off;
 11   101     0:00 nginx: worker process
 12   root    0:00 sh
 17   root    0:00 ps aux

Because the file system is shared, you can inspect the original container’s files via /proc/<PID>/root. For example, listing Nginx configuration files:

# ls /proc/6/root/etc/nginx
conf.d  koi-utf  mime.types  nginx.conf  uwsgi_params  fastcgi_params  koi-win  modules  scgi_params  win-utf

Conclusion

Ephemeral Containers greatly simplify debugging in Kubernetes, and sharing the process namespace enables advanced troubleshooting techniques. Teams can even automate workflows that react to readiness‑probe failures by spawning temporary containers to diagnose and remediate issues.