How to Enable and Use Kubernetes Ephemeral Containers for Debugging

What are Ephemeral Containers?

Ephemeral containers are temporary containers injected into a running pod for debugging. They share the Container.Spec fields with regular containers but prohibit fields such as ports, livenessProbe, readinessProbe, and resources. Once added, they cannot be modified or deleted.

Purpose

When the main container crashes or lacks debugging tools, kubectl exec may be ineffective. An ephemeral container provides an interactive shell without altering the pod spec.

Enable the Feature Gate

Add the flag --feature-gates=EphemeralContainers=true to the control‑plane components (kube‑apiserver, kube‑scheduler) and to the kubelet configuration, then restart the kubelet on all nodes:

# Edit kube-apiserver.yaml, kube-scheduler.yaml, and /etc/sysconfig/kubelet
--feature-gates=EphemeralContainers=true
systemctl restart kubelet

Create a Demo Pod

apiVersion: v1
kind: Pod
metadata:
  name: tomcat-test
  namespace: default
  labels:
    app: tomcat
spec:
  containers:
  - name: tomcat-java
    image: xianchao/tomcat-8.5-jre8:v1
    ports:
    - containerPort: 8080
    imagePullPolicy: IfNotPresent

Apply and verify:

kubectl apply -f pod-tomcat.yaml
kubectl get pods -n default

Add an Ephemeral Container

Use kubectl debug to inject a busybox container targeting the Tomcat container:

kubectl debug -it tomcat-test --image=busybox:1.28 --target=tomcat-java

Verify the new container appears in the pod description under the Ephemeral Containers section.

Update via Raw API

Because kubectl edit cannot modify ephemeralcontainers, prepare a JSON payload (e.g., a.json) and replace the subresource:

{
  "apiVersion": "v1",
  "kind": "EphemeralContainers",
  "metadata": {"name": "tomcat-test", "namespace": "default"},
  "ephemeralContainers": [{
    "name": "debugger",
    "image": "busybox",
    "command": ["sh"],
    "stdin": true,
    "tty": true,
    "targetContainerName": "tomcat-java",
    "imagePullPolicy": "IfNotPresent",
    "terminationMessagePolicy": "File"
  }]
}

kubectl replace --raw /api/v1/namespaces/default/pods/tomcat-test/ephemeralcontainers -f a.json

The API returns the created object, confirming the container is added.

Attach to the Ephemeral Container

kubectl attach -it -c debugger tomcat-test

Run debugging commands (e.g., ps -ef | grep tomcat). When you exit, the container terminates and cannot be re‑attached.

Limitations

Ephemeral containers cannot be deleted once added.

If the container exits, it cannot be re‑attached or restarted; a new container with a different name must be created.

These limitations are tracked in Kubernetes issue https://github.com/kubernetes/kubernetes/issues/84764.

Summary

Ephemeral containers enable on‑the‑fly debugging by injecting a temporary container into a running pod. Enable the feature gate, create the pod, add the container via kubectl debug or the raw API, and attach for inspection. Remember that the container cannot be deleted or restarted, so each debugging session may require a new ephemeral container.