How to Debug Kubernetes Pods Without Root Using Ephemeral Containers

If you run software on Kubernetes, you may need to debug aspects of a deployed pod. The usual approach of connecting to a running pod with kubectl exec -it podname -c containername -- bash often works, but two Kubernetes best‑practice rules limit its usefulness:

Containers should not run as root; they may use random UID.

Images should be minimal, sometimes distroless, lacking debugging tools.

When these practices are applied, kubectl exec either cannot connect or lands you in an unsuitable environment for debugging.

kubectl exec does not allow specifying user flags or capabilities; it inherits the settings from the container’s entrypoint.

Debugging Containers

Kubernetes provides a native debugging strategy using kubectl debug. This command launches a new temporary container in the same pod, allowing you to run it as any user and from any image. Because the debug container shares the pod’s node, it can share system resources with the target container.

For example, to inspect the CPU usage of a PostgreSQL container ( postcont) in pod postpod, where the pod does not run as root and the image lacks top or htop, you can run:

kubectl debug -it \
  --container=debug-container \
  --image=alpine \
  --target=postcont \
  postpod

The temporary container starts as root (the default for Alpine), letting you install htop ( apt add htop) and share the same process namespace with postcont, so you can view or even kill processes. Exiting the shell terminates the temporary container.

To keep the debug container in the same process namespace as postcont , you must use the --target flag, which is non‑optional. Detach from the temporary container with CTRL+P CTRL+D and later re‑attach using kubectl attach . kubectl debug also supports more advanced features such as copying pod specifications or launching a node‑debug pod.

Underlying Principle

kubectl debug

works by creating an ephemeral container that runs temporarily in an existing pod. Ephemeral containers are modeled as a sub‑resource of the pod, sharing the same process namespace and volume mounts as the target container, but they are not part of the pod spec.

Pods are intended to be disposable and immutable; the pod spec does not change, while ephemeral containers are added dynamically.

Volumes

The built‑in kubectl debug command can add a temporary container that shares the target container’s volume mounts. However, it cannot directly modify the target container’s filesystem because the debug pod’s filesystem is separate.

Read the target container’s spec.

Inject a temporary container with the same process namespace and volume mounts.

Since there is no direct kubectl command to create an ephemeral container, you can send a PATCH request to the Kubernetes API, often wrapped in a script or plugin such as kubectl-superdebug . After installing the script, you can run kubectl superdebug to achieve the same effect.

Non‑Native Methods

Kubernetes does not provide a way to exec as root unless the main process runs as root, nor a way to access another container’s root filesystem directly. If you have access to the Docker engine on the node, you can use docker exec --user to run a process as a chosen user.

Plugins like kubectl ssh or kubectl exec-user attempted to expose this, but modern runtimes (containerd, CRI‑O) no longer support the --user flag.

The kpexec tool works by entering the target container’s Linux namespaces from a privileged pod on the same node, allowing you to run commands with arbitrary UID/GID and install debugging tools. It is compatible with containerd and CRI‑O but may conflict with certain security configurations.

kpexec uses nsenter to execute inside namespaces; it works with the standard runc runtime but not with Kata Containers.

AI‑Assisted Debugging with Appilot

Appilot is an open‑source AI assistant for DevOps that leverages large language models to let users describe debugging tasks in natural language, simplifying Kubernetes debug workflows. The project is available at https://github.com/seal-io/appilot .

Overall, these native and non‑native techniques cover most debugging scenarios for running containers in Kubernetes.