Debugging Kubernetes Pods Without Root: Using kubectl debug and Alternatives

Using kubectl exec to Run Commands

If you run software on Kubernetes, you may want to debug aspects of your deployed application. For those used to virtual machines, a simple method is to connect to a running pod and execute a shell:

kubectl exec -it podname -c containername -- bash

While this often works, two Kubernetes best‑practice constraints limit exec’s usefulness:

Containers should not run as the root user; they may use minimal privileges or random UIDs.

Images should be as small as possible, sometimes using distroless images that lack a shell.

When these practices are applied, using kubectl exec may be impossible or land you in an unsuitable debugging environment.

kubectl exec does not allow specifying user flags or capabilities; it inherits the target container’s settings.

Debugging Containers

Kubernetes provides a native debugging strategy with kubectl debug. This command launches a new container inside the same pod, allowing you to run it as any user and from any image. The debug container shares the pod’s resources and can access the same process namespace.

For example, to inspect the CPU usage of a PostgreSQL container ( postcont) in pod postpod, which runs without root and lacks tools like top, you can run:

kubectl debug -it \
  --container=debug-container \
  --image=alpine \
  --target=postcont \
  postpod

This starts a root‑owned Alpine container; you can install htop ( apt add htop) and share the process namespace with postcont, letting you view or kill its processes. Exiting the shell terminates the temporary container.

To share the same process namespace with postcont , use the non‑optional --target flag. Detach from the temporary container with CTRL+P CTRL+D and later re‑attach using kubectl attach . kubectl debug also supports copying pods, modifying start commands, or launching a node‑access pod.

Underlying Mechanism

kubectl debug

creates an ephemeral container that runs alongside existing containers in the pod. Although ordinary and ephemeral containers are similar, pods are immutable; therefore, an ephemeral container is modeled as a sub‑resource rather than part of the pod spec.

Volumes

The built‑in kubectl debug command can add a temporary container that shares the target container’s process namespace and volume mounts. However, the debug pod’s filesystem is isolated from the target container’s filesystem. To copy specifications or mount the same volumes, the process involves reading the target container’s spec and patching the pod via the Kubernetes API (often using kubectl proxy).

A script implementation is available at kubectl‑superdebug . After placing the script in your PATH, you can run kubectl superdebug to create an ephemeral debug container and optionally copy environment variables.

Non‑Native Methods

Kubernetes does not provide a built‑in way to exec as root unless the main process runs as root, nor does it allow direct access to a container’s root filesystem from another container. If you have direct node access (e.g., Docker engine), you can use docker exec --user to run commands as a chosen user.

Plugins like kubectl ssh and kubectl exec-user attempted similar functionality, but modern runtimes such as containerd and CRI‑O no longer support the required --user flag.

The kpexec tool creates a privileged pod on the same node, enters the target container’s namespaces, and streams command output to your terminal. It can also overlay debugging tools onto the container’s filesystem. Unlike kubectl exec, kpexec can run commands with different UID/GID and capabilities, and works with containerd and CRI‑O, though it may conflict with cluster security policies.

kpexec uses nsenter to execute commands directly in namespaces; it is compatible with runc but not with Kata Containers.

Appilot: Conversational K8s Debugging

Appilot is an open‑source AI assistant for DevOps that leverages large language models to let users issue natural‑language commands for Kubernetes debugging and management. It can be integrated into any platform, enabling simple, AI‑driven interactions with Kubernetes.

Project URL: https://github.com/seal-io/appilot

(© Original author, please delete if infringing)