Mastering Kubernetes Debugging with Ephemeral Containers: A Practical Guide

In the past, debugging containers on Kubernetes often required restarting pods or embedding sleep commands, especially when using distroless images, leaving operators helpless.

kubectl-debug was once a popular solution, consisting of an agent and debug-tools, but its development has stalled and it does not work with Containerd runtimes.

Modern Kubernetes offers a better approach: Ephemeral Containers, introduced as an alpha feature in v1.16 and integrated into kubectl from v1.18 onward.

Ephemeral Containers

Ephemeral Containers are temporary containers that provide a fault‑diagnosis tool without relying on tools baked into the original image and without needing direct node login.

Out‑of‑the‑box platform tool

No dependency on utilities inside the container image

Access via the Kubernetes API instead of SSH

They are not intended for Windows nodes and must be enabled with the feature gate:

--feature-gates=EphemeralContainers=true

Before v1.20 the kubectl debug command was in alpha; using a client version 1.20+ provides a smoother experience.

1. Pod Troubleshooting

Debug a pod with a simple command: kubectl debug mypod -it --image=busybox Specify a custom name for the debug container: kubectl debug mypod -c debugger --image=busybox Ephemeral Containers can also be used for batch operations, such as running a security scanner across a namespace:

for pod in $(kubectl get -o name pod); do
    kubectl debug --image security/pod_scanner -p $pod /scanner.sh
done

2. Pod Troubleshooting by Copy

If the cluster does not have the Ephemeral Containers feature enabled, a copy‑to mode can be used, which creates a new pod with a sidecar debug container. Important flags include:

--copy-to   Specify the new pod name
--replace=true   Delete the original pod
--same-node=true   Schedule on the same node
--share-processes=true   Share the PID namespace

Example:

kubectl debug mypod -it \
  --container=debug \
  --image=busybox \
  --copy-to=my-debugger \
  --same-node=true \
  --share-processes=true

3. Node Troubleshooting

Ephemeral Containers can also debug a worker node by creating a pod on the target node with privileged modes (hostIPC, hostNetwork, hostPID) and mounting the node’s root filesystem under /host:

kubectl debug node/mynode -it --image=busybox

Debug Image

A well‑equipped debug image simplifies troubleshooting. Below is a Dockerfile that builds a multi‑stage image with common tools and grpcurl:

FROM golang:alpine as grpcurl
RUN apk update && \
    apk add --virtual build-dependencies git && \
    apk add bash curl jq && \
    go get -u github.com/fullstorydev/grpcurl && \
    go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest

FROM alpine:latest
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories && \
    apk update && \
    apk add --no-cache vim bash tcpdump curl wget strace mysql-client iproute2 redis jq iftop tzdata tar nmap bind-tools htop && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN wget -O /usr/bin/httpstat https://github.com/davecheney/httpstat/releases/download/v1.0.0/httpstat-linux-amd64-v1.0.0 && \
    chmod +x /usr/bin/httpstat
COPY --from=grpcurl /go/bin/grpcurl /usr/bin/grpcurl
ENV TZ=Asia/Shanghai LC_ALL=C.UTF-8 LANG=C.UTF-8 LANGUAGE=C.UTF-8
ENTRYPOINT ["/bin/bash"]

The resulting image includes tools such as vim, bash, tcpdump, curl, wget, strace, mysql-client, redis, and more (illustrated below).

Conclusion

The article explains how the Ephemeral Containers feature, introduced as alpha in Kubernetes v1.16 and matured in v1.18, enables container and even host debugging without the limitations of older tools like kubectl‑debug. While permissions and privileged modes require careful handling, Ephemeral Containers offer a powerful, native solution for Kubernetes troubleshooting.