Why Kubernetes Ephemeral Containers Make Debugging Pods a Breeze

When the container concept was introduced, the key principle was minimalism: only the processes needed to run should be present, nothing else. This is why early Docker articles highlighted that an SSH daemon ( sshd) isn’t required inside a container.

However, minimal containers often lack essential utilities such as curl, wget, ifconfig, ip, or tcpdump. When troubleshooting, the absence of these tools can be frustrating, especially with third‑party images where rebuilding the image to add them isn’t convenient.

One early idea was to add a --from-host flag to kubectl exec so the kubelet could invoke nsenter on the host and run binaries directly, bypassing the container’s limitations. While appealing, the author later shifted focus to AI platforms and did not pursue this.

Recently, Kubernetes introduced “ephemeral containers” (temporary containers), a feature that elegantly solves the problem. The platform adds a kubectl debug command that creates a temporary container sharing the target pod’s namespaces, allowing any required tools to be provided via a custom image.

How It Works

The process is illustrated in the diagram below (red container represents the temporary container). By specifying an image, you control which command‑line utilities are available inside the temporary container.

For example, to debug a container named app in a pod, you can run:

kubectl debug -it -c debugger --target=app --image=busybox <POD_NAME>

The --target flag selects the specific container to debug when a pod has multiple containers.

You can also launch a temporary container on a node without installing any tools on the node itself: kubectl debug node/mynode -it --image=ubuntu This approach keeps the host node minimal while providing full debugging capabilities via the temporary container, demonstrating a thoughtful and extensible design.

Original source: https://bbs.huaweicloud.com/blogs/402638