How to Extend Knative with a Custom Admission Webhook for Volume Injection

Background

Serverless platforms such as Knative rely on Kubernetes admission control to validate and mutate resources before they are persisted. Knative only accepts a limited set of volume types (Secret, ConfigMap, Projected, EmptyDir), causing failures when other volume types are needed.

Problem Statement

Modifying Knative’s built‑in webhook code is invasive and hard to maintain. A lightweight, non‑intrusive solution is required to add arbitrary volumes to pods created by Knative services without changing Knative’s source.

Solution Overview

A custom MutatingAdmissionWebhook is deployed alongside Knative. It reads serialized volume definitions from pod annotations, injects the corresponding volumes and volumeMounts into the pod spec, and adds a label to indicate successful mutation.

1. Admission Control Basics

When an API request reaches the kube-apiserver, it passes through authentication, then optional MutatingAdmissionWebhook and ValidatingAdmissionWebhook stages before being stored in etcd. A webhook can modify the object by returning a JSON‑Patch.

2. Dynamic Admission Webhook Requirements

Kubernetes version ≥ v1.16 (or v1.9 for the beta API).

Both MutatingAdmissionWebhook and ValidatingAdmissionWebhook controllers must be enabled.

The admissionregistration.k8s.io/v1 or v1beta1 API must be available.

3. Implementing the Webhook in Go

The webhook handler reads the incoming AdmissionReview, unmarshals the Pod, and performs three main actions:

Append a new Volume (sourced from a ConfigMap) to pod.Spec.Volumes.

Append a corresponding VolumeMount to every container.

Add a label hello-added=OK via a JSON‑Patch.

func (app *App) HandleMutate(w http.ResponseWriter, r *http.Request) {
    admissionReview := &admissionv1.AdmissionReview{}
    if err := readJSON(r, admissionReview); err != nil { app.HandleError(w, r, err); return }
    pod := &corev1.Pod{}
    if err := json.Unmarshal(admissionReview.Request.Object.Raw, pod); err != nil { app.HandleError(w, r, fmt.Errorf("unmarshal to pod: %v", err)); return }
    // add volume
    pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{...})
    // add volumeMount to each container
    for i := range pod.Spec.Containers {
        pod.Spec.Containers[i].VolumeMounts = append(pod.Spec.Containers[i].VolumeMounts, corev1.VolumeMount{...})
    }
    // build JSON patch
    patch := []JSONPatchEntry{{OP:"add", Path:"/metadata/labels/hello-added", Value:[]byte(`"OK"`)}}
    // marshal and respond
    patchBytes, _ := json.Marshal(patch)
    admissionResponse := &admissionv1.AdmissionResponse{UID: admissionReview.Request.UID, Allowed: true, Patch: patchBytes, PatchType: func() *admissionv1.PatchType { pt := admissionv1.PatchTypeJSONPatch; return &pt }()}
    resp := &admissionv1.AdmissionReview{Response: admissionResponse}
    jsonOk(w, resp)
}

4. TLS Certificate Generation

The webhook server must serve HTTPS. A helper Bash script creates a CSR, signs it via the Kubernetes CertificateSigningRequest API, and stores the key and certificate in a secret:

#!/bin/bash
set -e
while [[ $# -gt 0 ]]; do
  case $1 in
    --service) service="$2"; shift;;
    --secret) secret="$2"; shift;;
    --namespace) namespace="$2"; shift;;
  esac
  shift
done
# generate key, CSR, approve, extract cert, create secret …

5. Deploying the Webhook

Two Kubernetes manifests are required:

A Deployment that runs the Go binary and mounts the TLS secret.

A Service exposing the webhook on port 443.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfaas-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfaas-webhook
  template:
    metadata:
      labels:
        app: nfaas-webhook
    spec:
      containers:
      - name: nfaas-webhook-container
        image: nfaas-webhook:latest
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
          readOnly: true
      volumes:
      - name: webhook-certs
        secret:
          secretName: nfaas-webhook-certs

apiVersion: v1
kind: Service
metadata:
  name: nfaas-webhook
spec:
  ports:
  - port: 443
    targetPort: 8088
    protocol: TCP
  selector:
    app: nfaas-webhook
  type: ClusterIP

6. Registering the Webhook

A MutatingWebhookConfiguration points to the service and limits the scope to the target namespace and Knative‑related pods:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: nfaas-webhook
webhooks:
- name: mutating.nfaas.com
  admissionReviewVersions: ["v1"]
  sideEffects: NoneOnDryRun
  clientConfig:
    service:
      name: nfaas-webhook
      namespace: default
      path: "/pod"
    caBundle: ${CA_BUNDLE}
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    scope: "*"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: nfaas-demo
  objectSelector:
    matchExpressions:
    - key: serving.knative.dev/service
      operator: Exists

7. Verification

Create a pod with the label hello=true and observe that the webhook injects hello.txt from the ConfigMap and adds the hello-added=OK label. A second pod without the label shows no injection, confirming selective mutation.

# Create a test pod
kubectl run busybox-1 --image=busybox --restart=Never -l=app=busybox,hello=true -- sleep 3600
# Verify file injection
kubectl exec busybox-1 -it -- cat /etc/config/hello.txt
# Verify label
kubectl get pod -l=app=busybox -L=hello-added

The tutorial demonstrates a practical, non‑intrusive way to extend Knative’s capabilities by leveraging Kubernetes dynamic admission control, providing reusable code snippets, deployment manifests, and a clear verification workflow.