Debugging Running Pods in Kubernetes Without Root Access

Limitations of kubectl exec

When a pod follows security‑best‑practice guidelines—running containers as non‑root users (often with random UID) and using minimal or distroless images— kubectl exec -it <pod> -c <container> -- bash may either fail to attach or drop you into a shell that lacks debugging utilities such as top or htop. The command inherits the target container’s user and capabilities, so you cannot override them.

Native debugging with kubectl debug

kubectl debug

creates an ephemeral container inside the same pod. The temporary container can use any image, runs as root by default, and shares the pod’s process namespace and volume mounts with the target container. This makes it possible to install debugging tools on‑the‑fly and inspect or kill processes of the original container.

kubectl debug -it \
  --container=debug-container \
  --image=alpine \
  --target=postcont \
  postpod

In the example above, the pod postpod contains a PostgreSQL container postcont that runs without root and lacks htop. The debug container starts as root (Alpine’s default), allowing you to run apk add htop (or apt-get update && apt-get install -y htop on Debian‑based images) and view the full process list because both containers share the same PID namespace. Exiting the debug shell automatically removes the temporary container.

Detach from the debug session without terminating the container by pressing CTRL+P CTRL+D , then reconnect later with kubectl attach -c debug-container <pod> .

How kubectl debug works

The command injects an ephemeral container as a sub‑resource of the pod. Ephemeral containers are not part of the immutable pod spec, so they can be added to a running pod without recreating it. The injected container is configured to:

Share the target container’s process namespace ( shareProcessNamespace: true).

Mount the same volumes as the target container.

Volume mounting and filesystem access

While the debug container can see the same volume mounts, its own root filesystem is isolated from the target container’s image layers. To inspect files inside the target container’s image you must rely on the shared volumes or copy files out of the container.

Creating an ephemeral container manually

There is no dedicated kubectl sub‑command for arbitrary ephemeral‑container creation. The typical workflow is:

Run kubectl proxy to expose the Kubernetes API locally.

Send a PATCH request to

/api/v1/namespaces/<ns>/pods/<pod>/ephemeralcontainers

with a JSON payload describing the debug container (image, command, securityContext, volumeMounts, etc.).

Community scripts wrap this process. One example is the kubectl-superdebug script, which can be installed in $PATH and invoked as kubectl superdebug to automate the PATCH request and optionally copy environment variables from the target container.

Repository: https://github.com/JonMerlevede/kubectl-superdebug

Non‑native alternatives

If you have node‑level access (e.g., direct Docker engine access), you can run: docker exec --user 0 <container-id> bash to start a root shell inside the container. This approach bypasses Kubernetes entirely and therefore works only when you control the node.

Several kubectl plugins attempt to provide similar functionality ( kubectl ssh, kubectl exec-user), but modern runtimes such as containerd and CRI‑O have removed support for the --user flag, limiting their usefulness.

The kpexec tool launches a privileged pod on the same node, uses nsenter to enter the target container’s Linux namespaces, and streams a shell back to the client. It can also overlay additional debugging binaries onto the target filesystem. kpexec works with the runc runtime but is incompatible with runtimes like Kata Containers and may conflict with cluster security policies.

kpexec relies on nsenter to execute inside the container’s namespaces, making it compatible with runc but not with all container runtimes.

AI‑assisted debugging with Appilot

Appilot is an open‑source AI assistant for DevOps that accepts natural‑language commands to perform Kubernetes debugging tasks (e.g., “show me the logs of pod X” or “launch a debug container for pod Y”). It can be integrated into CI/CD pipelines or chat platforms to streamline troubleshooting.

Project repository: https://github.com/seal-io/appilot