How to Enable and Use Ephemeral Containers in Kubernetes for Debugging

What Are Ephemeral Containers?

Ephemeral containers are special temporary containers in Kubernetes used for debugging. Unlike regular containers, they do not have guaranteed resources, cannot be automatically restarted, and lack port, livenessProbe, readinessProbe, and resources configurations.

They are defined using the ephemeralcontainers field in the pod spec.

They cannot be added via kubectl edit because they are not part of pod.spec.

Typical Use Cases

When a container crashes or the container image lacks debugging utilities, kubectl exec becomes ineffective. In such cases, an Ephemeral container provides an interactive environment for troubleshooting.

Enabling the EphemeralContainers Feature Gate

To use Ephemeral containers, the feature gate must be enabled on the control‑plane components and the kubelet.

# Edit kube-apiserver manifest
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --feature-gates=EphemeralContainers=true
    ...

# Edit kube-scheduler manifest
apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --feature-gates=EphemeralContainers=true
    ...

# Add the flag to kubelet configuration on all nodes
KUBELET_EXTRA_ARGS="--feature-gates=EphemeralContainers=true"

After modifying the manifests and kubelet config, restart the kubelet on each node:

# systemctl restart kubelet

Creating a Sample Tomcat Pod

apiVersion: v1
kind: Pod
metadata:
  name: tomcat-test
  namespace: default
  labels:
    app: tomcat
spec:
  containers:
  - name: tomcat-java
    image: xianchao/tomcat-8.5-jre8:v1
    ports:
    - containerPort: 8080
    imagePullPolicy: IfNotPresent

Apply the pod definition:

# kubectl apply -f pod-tomcat.yaml

Adding an Ephemeral Container

Use kubectl debug to inject a temporary container targeting the existing Tomcat container:

# kubectl debug -it tomcat-test --image=busybox:1.28 --target=tomcat-java

The command creates a debug container (e.g., debugger-6m2s8) and drops you into an interactive shell.

Verify the Ephemeral container is attached:

# kubectl describe pod tomcat-test

Updating an Ephemeral Container via the Raw API

Delete and recreate the original pod to ensure a clean state, then prepare a JSON payload describing the Ephemeral container:

{
  "apiVersion": "v1",
  "kind": "EphemeralContainers",
  "metadata": {"name": "tomcat-test", "namespace": "default"},
  "ephemeralContainers": [{
    "name": "debugger",
    "image": "busybox",
    "command": ["sh"],
    "stdin": true,
    "tty": true,
    "targetContainerName": "tomcat-java",
    "imagePullPolicy": "IfNotPresent",
    "terminationMessagePolicy": "File"
  }]
}

Apply the payload with a raw replace request:

# kubectl replace --raw /api/v1/namespaces/default/pods/tomcat-test/ephemeralcontainers -f a.json

The API returns the created Ephemeral container object, confirming the operation succeeded.

Attaching to the Ephemeral Container

Once the container is running, attach to it for interactive debugging: # kubectl attach -it -c debugger tomcat-test From the attached shell you can run typical commands, such as inspecting the Tomcat process: / # ps -ef | grep tomcat When finished, exit the shell with exit.