Master Kubernetes Debugging with kubectl debug and Ephemeral Containers

Why Traditional Debugging Often Fails

Developers and DevOps engineers regularly use kubectl logs or kubectl describe pod to locate problems, but these tools can be insufficient when containers are built without a shell (e.g., Distroless images) or when kubectl exec cannot attach.

Introducing kubectl debug and EphemeralContainers

The kubectl debug command (added in v1.18) injects a special EphemeralContainer into a running pod, allowing inspection without modifying the original pod spec. EphemeralContainers are temporary, read‑only sub‑resources used solely for debugging; they lack fields such as ports and resources and are not part of the pod’s initial definition.

Because EphemeralContainers are still in Alpha, they are disabled by default and must be enabled via the EphemeralContainers feature gate.

Configuring the Feature Gate

When using kubeadm you can enable the gate by adding the following to the cluster configuration:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.2
apiServer:
  extraArgs:
    feature-gates: EphemeralContainers=true

For a KinD cluster, a minimal config looks like:

# File: config.yaml
# Run:  kind create cluster --config ./config.yaml --name kind --image=kindest/node:v1.20.2
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  EphemeralContainers: true
nodes:
- role: control-plane

After the cluster starts, verify the feature is active:

$ kubectl explain pod.spec.ephemeralContainers
KIND:      Pod
VERSION:   v1
RESOURCE:  ephemeralContainers <[]Object>
DESCRIPTION:
  List of ephemeral containers run in this pod....

Basic Usage of kubectl debug

Create a simple pod and attach a debugging container:

$ kubectl run some-app --image=k8s.gcr.io/pause:3.1 --restart=Never
$ kubectl debug -it some-app --image=busybox --target=some-app
Defaulting debug container name to debugger-xxxxx.

Describing the pod now shows an EphemeralContainers section with the injected container.

Process Namespace Sharing

When the original container lacks debugging tools, use --share-processes to share the process namespace and --copy-to to create a new pod that includes the temporary container:

$ kubectl run some-app --image=nginx --restart=Never
$ kubectl debug -it some-app --image=busybox --share-processes --copy-to=some-app-debug

Inside the debug pod you can inspect the original container’s processes and files via /proc:

# ps ax
PID   USER   TIME   COMMAND
1     root   0:00   /pause
8     root   0:00   /usr/bin/nginx -g daemon off;
# cat /proc/8/root/etc/nginx/conf.d/default.conf
server {
    listen 80;
    listen [::]:80;
    server_name localhost;
    ...
}

Debugging Nodes

kubectl debug can also create a pod on a specific node, giving you host‑level access. After creating the pod, use chroot /host to enter the node’s root filesystem:

$ kubectl debug node/kind-control-plane -it --image=ubuntu
Creating debugging pod node-debugger-xxxxx with container debugger on node kind-control-plane.
root@kind-control-plane:/# chroot /host
# head /etc/kubernetes/kubeadm.conf
apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  extraArgs:
    feature-gates: EphemeralContainers=true
  runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta2
clusterName: kind
controlPlaneEndpoint: kind-control-plane:6443

When Pods Crash Immediately

If a pod crashes before you can attach, create a copy with a different entrypoint using --copy-to and --container:

$ kubectl debug crashing-app -it --copy-to=crashing-app-debug --container=crashing-app -- sh
# uid=0(root) gid=0(root) groups=0(root)

The new pod stays running, allowing you to investigate the filesystem and environment.

Conclusion

Using kubectl debug and EphemeralContainers provides a powerful, flexible way to troubleshoot Kubernetes workloads, nodes, and crashing pods without rebuilding images. Even though the feature is still in Alpha, enabling the feature gate unlocks capabilities that can save significant time and cost.

Original article: https://towardsdatascience.com/the-easiest-way-to-debug-kubernetes-workloads-ff2ff5e3cc75