How to Build a Kubernetes Admission Controller to Block Namespace Deletion

Introduction

Yesterday a friend accidentally deleted a namespace, causing the whole business to stop. He asked for a way to forbid namespace deletion. Since Kubernetes admission does not provide this controller out‑of‑the‑box, we need to develop a custom admission controller.

Theory

Admission controllers sit in the API server and intercept requests before they are persisted. They are commonly used for authentication and authorization. Two special controllers are MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

MutatingAdmissionWebhook: can modify the request object, e.g., Istio injects sidecars.

ValidatingAdmissionWebhook: validates the request object.

The admission control flow is as follows:

When an API request arrives, mutating and validating controllers concurrently call the external webhooks listed in the configuration. If all webhooks approve, the request continues; if any webhook rejects, the request is terminated and the first rejection reason is returned; if an error occurs, the request may be terminated or the webhook ignored.

Admission controllers are configured via API server startup parameters. A cluster usually enables a default set of admission controllers; without them the cluster is “bare”.

Implementation

Logic Overview

A webhook is a standard HTTP service that receives an AdmissionReview object, processes it, and returns an AdmissionReview containing the result.

AdmissionReview struct (simplified):

type AdmissionReview struct {
    metav1.TypeMeta `json:",inline"`
    Request  *AdmissionRequest  `json:"request,omitempty"`
    Response *AdmissionResponse `json:"response,omitempty"`
}

Server Code (main.go)

package main

import (
    "context"
    "flag"
    "github.com/joker-bai/validate-namespace/http"
    log "k8s.io/klog/v2"
    "os"
    "os/signal"
    "syscall"
)

var (
    tlscert, tlskey, port string
)

func main() {
    flag.StringVar(&tlscert, "tlscert", "/etc/certs/cert.pem", "Path to the TLS certificate")
    flag.StringVar(&tlskey, "tlskey", "/etc/certs/key.pem", "Path to the TLS key")
    flag.StringVar(&port, "port", "8443", "The port to listen")
    flag.Parse()

    server := http.NewServer(port)
    go func() {
        if err := server.ListenAndServeTLS(tlscert, tlskey); err != nil {
            log.Errorf("Failed to listen and serve: %v", err)
        }
    }()

    log.Infof("Server running in port: %s", port)

    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
    <-signalChan

    log.Info("Shutdown gracefully...")
    if err := server.Shutdown(context.Background()); err != nil {
        log.Error(err)
    }
}

HTTP Handler

package http

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"

    "github.com/douglasmakey/admissioncontroller"
    admission "k8s.io/api/admission/v1beta1"
    meta "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/serializer"
    log "k8s.io/klog/v2"
)

type admissionHandler struct {
    decoder runtime.Decoder
}

func newAdmissionHandler() *admissionHandler {
    return &admissionHandler{
        decoder: serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer(),
    }
}

func (h *admissionHandler) Serve(hook admissioncontroller.Hook) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        if r.Method != http.MethodPost {
            http.Error(w, fmt.Sprint("invalid method only POST requests are allowed"), http.StatusMethodNotAllowed)
            return
        }
        if contentType := r.Header.Get("Content-Type"); contentType != "application/json" {
            http.Error(w, fmt.Sprint("only content type 'application/json' is supported"), http.StatusBadRequest)
            return
        }
        body, err := io.ReadAll(r.Body)
        if err != nil {
            http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)
            return
        }
        var review admission.AdmissionReview
        if _, _, err := h.decoder.Decode(body, nil, &review); err != nil {
            http.Error(w, fmt.Sprintf("could not deserialize request: %v", err), http.StatusBadRequest)
            return
        }
        if review.Request == nil {
            http.Error(w, "malformed admission review: request is nil", http.StatusBadRequest)
            return
        }
        result, err := hook.Execute(review.Request)
        if err != nil {
            log.Error(err)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        admissionResponse := admission.AdmissionReview{
            Response: &admission.AdmissionResponse{
                UID:     review.Request.UID,
                Allowed: result.Allowed,
                Result:  &meta.Status{Message: result.Msg},
            },
        }
        res, err := json.Marshal(admissionResponse)
        if err != nil {
            log.Error(err)
            http.Error(w, fmt.Sprintf("could not marshal response: %v", err), http.StatusInternalServerError)
            return
        }
        log.Infof("Webhook [%s - %s] - Allowed: %t", r.URL.Path, review.Request.Operation, result.Allowed)
        w.WriteHeader(http.StatusOK)
        w.Write(res)
    }
}

func healthz() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("ok"))
    }
}

Hook Registration

package namespace

import (
    "github.com/douglasmakey/admissioncontroller"
    log "k8s.io/klog/v2"
    "k8s.io/api/admission/v1beta1"
)

func NewValidationHook() admissioncontroller.Hook {
    return admissioncontroller.Hook{
        Delete: validateDelete(),
    }
}

func validateDelete() admissioncontroller.AdmitFunc {
    return func(r *v1beta1.AdmissionRequest) (*admissioncontroller.Result, error) {
        if r.Kind.Kind == "Namespace" {
            log.Info("You cannot delete namespace: ", r.Name)
            return &admissioncontroller.Result{Allowed: false}, nil
        }
        return &admissioncontroller.Result{Allowed: true}, nil
    }
}

Deployment and Testing

Docker Image

FROM golang:1.17.5 AS build-env
ENV GOPROXY https://goproxy.cn
ADD . /go/src/app
WORKDIR /go/src/app
RUN go mod tidy
RUN cd cmd && GOOS=linux GOARCH=amd64 go build -v -a -ldflags '-extldflags "-static"' -o /go/src/app/app-server /go/src/app/cmd/main.go

FROM registry.cn-hangzhou.aliyuncs.com/coolops/ubuntu:22.04
ENV TZ=Asia/Shanghai
COPY --from=build-env /go/src/app/app-server /opt/app-server
WORKDIR /opt
EXPOSE 80
CMD ["./app-server"]

Certificate Generation Script

#!/bin/bash
set -e
# (script omitted for brevity)

Kubernetes Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: validate-delete-namespace
  labels:
    app: validate-delete-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: validate-delete-namespace
  template:
    metadata:
      labels:
        app: validate-delete-namespace
    spec:
      containers:
      - name: server
        image: registry.cn-hangzhou.aliyuncs.com/coolops/validate-delete-namespace:latest
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8443
            scheme: HTTPS
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: tls-certs
          mountPath: /etc/certs
          readOnly: true
      volumes:
      - name: tls-certs
        secret:
          secretName: validate-delete-namespace-tls
---
apiVersion: v1
kind: Service
metadata:
  name: validate-delete-namespace
spec:
  selector:
    app: validate-delete-namespace
  ports:
  - port: 443
    targetPort: 8443

ValidatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validate-delete-namespace
webhooks:
- name: validate-delete-namespace.default.svc.cluster.local
  clientConfig:
    service:
      namespace: default
      name: validate-delete-namespace
      path: "/validate/delete-namespace"
    caBundle: "${CA_BUNDLE}"
  rules:
  - operations: ["DELETE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["namespaces"]
  failurePolicy: Ignore

Testing

Create a namespace and attempt to delete it. The webhook rejects the deletion, leaving the namespace intact.

# kubectl create ns joker
# kubectl delete ns joker
Error from server: admission webhook "validate-delete-namespace.default.svc.cluster.local" denied the request without explanation

Logs show the rejection message:

2022/06/24 17:43:34 You cannot delete namespace:  joker
I0624 17:43:34.664945 1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

References

https://www.qikqiak.com/post/k8s-admission-webhook

https://github.com/douglasmakey/admissioncontroller

https://mritd.com/2020/08/19/write-a-dynamic-admission-control-webhook/