K8s Deletion Defense: Dual‑Ring Protection with Auth and Validation

Why Kubernetes Deletion Needs Its Own Protection System

In most teams the delete API call is treated as a simple permission check, but production‑grade deletion requires answering four questions: who can delete, whether the resource is critical, if the request occurs within an approved change window, and what the cascade impact would be. Relying solely on RBAC or native admission results in a crude "RBAC allows, so delete succeeds" outcome.

Goals of a Production‑Level Deletion Guard

The system must provide layered decision‑making, programmability, observability, gradual rollout, and an escape hatch. Seven concrete capabilities are defined:

Authorization ring – fast "can I delete?" checks.

Validation ring – deeper "should I delete?" analysis.

Buffer layer – asynchronous cleanup and cool‑down.

Audit layer – post‑mortem traceability.

High availability of control‑plane components.

High concurrency handling.

Extensibility for new resources, approval systems, and risk models.

Core Design: Authorization Ring + Validation Ring + Buffer + Audit

┌──────────────────────────────┐
                │ kubectl / CI / platform API   │
                └──────────────┬───────────────┘
                               │
                               ▼
                ┌──────────────────────────────┐
                │ Kubernetes API Server          │
                └──────────────┬───────────────┘
                               │
               ┌───────────────┴───────────────┐
               │                               │
        ┌──────▼─────┐                 ┌─────▼─────┐
        │ Auth Ring   │                 │ Validate   │
        │ (OPA Gatekeeper)│             │ Ring (Webhook)│
        └──────┬─────┘                 └─────┬─────┘
               │                               │
               └───────────────┬───────────────┘
                               ▼
                ┌──────────────────────────────┐
                │ Buffer Layer (Finalizer)      │
                └──────────────┬───────────────┘
                               │
                               ▼
                ┌──────────────────────────────┐
                │ etcd / Controllers / GC       │
                └──────────────┬───────────────┘
                               │
                               ▼
                ┌──────────────────────────────┐
                │ Audit Layer (Kafka / S3 / ES) │
                └──────────────────────────────┘

Why a Dual‑Ring Instead of a Single Webhook

Putting all logic in one webhook leads to mixed responsibilities, unpredictable latency, and difficult iteration. Splitting into an Authorization Ring (fast policy evaluation via OPA Gatekeeper) and a Validation Ring (expensive dependency analysis, approval checks) isolates concerns and keeps the synchronous path short.

Authorization Ring Details

The ring answers "can I delete?" using lightweight checks:

Is the resource marked as protected?

Is the requester allowed in the current environment?

Is the request inside a configured change window?

Is a required ticket present?

Does the request come from a trusted CI/CD service?

These rules are expressed as an OPA ConstraintTemplate (see code snippet below) and can be version‑controlled via GitOps.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdeletionguard
spec:
  crd:
    spec:
      names:
        kind: K8sDeletionGuard
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdeletionguard
        default is_delete = false
        is_delete { input.review.operation == "DELETE" }
        protected_kind(kind) { kind == input.parameters.protectedKinds[_] }
        protected_namespace(ns) { ns == input.parameters.protectedNamespaces[_] }
        in_restricted_hour {
          hour := time.clock(time.now_ns())[0]
          hour >= input.parameters.restrictedHours.start
          hour < input.parameters.restrictedHours.end
        }
        violation[{"msg": msg}] {
          is_delete
          protected_kind(input.review.kind.kind)
          in_restricted_hour
          msg := sprintf("delete denied: %v is in restricted hours", [input.review.kind.kind])
        }
        violation[{"msg": msg}] {
          is_delete
          protected_namespace(input.review.namespace)
          not has_required_ticket
          msg := sprintf("delete denied: protected resource %v/%v requires ticket %v", [input.review.kind.kind, input.review.name, input.parameters.ticketAnnotation])
        }

Validation Ring Details

The validation webhook performs deep analysis that cannot be expressed statically:

Count of active Pods owned by the target.

Presence of external traffic entry points (Ingress, Service).

Bound PersistentVolumeClaims.

Current business hour status.

Approval ticket existence and status.

A scoring model aggregates these signals into a risk level (LOW, MEDIUM, HIGH, CRITICAL). Example scoring logic is shown in the Go function BuildRiskReport:

func BuildRiskReport(target ResourceRef, level ProtectionLevel, deps []Dependency, now time.Time) *RiskReport {
    report := &RiskReport{Target: target, ProtectionLevel: level, Dependencies: deps}
    switch level {
    case ProtectionCritical:
        report.Score += 40
        report.Reasons = append(report.Reasons, "resource is marked critical")
    case ProtectionHigh:
        report.Score += 20
        report.Reasons = append(report.Reasons, "resource is marked high")
    }
    if target.Kind == "Namespace" { report.Score += 30; report.Reasons = append(report.Reasons, "namespace deletion has broad blast radius") }
    for _, dep := range deps {
        switch dep.Category {
        case "pod": report.AffectedPodCount++
        case "ingress": report.AffectedIngresses++
        case "pvc": report.BoundPVCCount++
        }
    }
    if report.AffectedPodCount >= 20 { report.Score += 20; report.Reasons = append(report.Reasons, "more than 20 active pods affected") } else if report.AffectedPodCount > 0 { report.Score += 10; report.Reasons = append(report.Reasons, "active pods are affected") }
    if report.AffectedIngresses > 0 { report.Score += 15; report.Reasons = append(report.Reasons, "public traffic entry is affected"); report.EstimatedTrafficQPS = float64(report.AffectedPodCount) * 120 }
    if report.BoundPVCCount > 0 { report.Score += 20; report.Reasons = append(report.Reasons, "persistent data may be impacted") }
    hour := now.Hour()
    if hour >= 10 && hour < 22 { report.Score += 10; report.Reasons = append(report.Reasons, "operation happens in business hours") }
    switch {
    case report.Score >= 80: report.Level = RiskCritical
    case report.Score >= 50: report.Level = RiskHigh
    case report.Score >= 25: report.Level = RiskMedium
    default: report.Level = RiskLow
    }
    return report
}

Decision Logic

The validator combines the risk report with policy configuration (allowed windows, required tickets, approval count) to produce a Decision object. High‑risk deletions may be allowed only with a cool‑down period, while low‑risk ones are permitted immediately.

func (v *Validator) decide(ctx context.Context, req *admissionv1.AdmissionRequest, profile *ProtectionProfile, report *RiskReport, oldObject metav1.Object) (*Decision, error) {
    if !inAllowedWindow(profile.AllowedTimeWindows, time.Now()) && report.Level != RiskLow {
        return &Decision{Allowed: false, Code: "WINDOW_BLOCKED", Message: fmt.Sprintf("delete denied: not in allowed change window, risk=%s score=%d", report.Level, report.Score), RiskReport: report}, nil
    }
    if profile.RequireTicket || report.Level == RiskHigh || report.Level == RiskCritical {
        ticketID := oldObject.GetAnnotations()[TicketAnnotation]
        if ticketID == "" {
            return &Decision{Allowed: false, Code: "MISSING_TICKET", Message: "delete denied: approval ticket is required", RiskReport: report}, nil
        }
        ticket, err := v.tickets.ValidateTicket(ctx, ticketID)
        if err != nil { return nil, fmt.Errorf("validate ticket: %w", err) }
        if ticket.Status != "APPROVED" {
            return &Decision{Allowed: false, Code: "TICKET_NOT_APPROVED", Message: fmt.Sprintf("delete denied: ticket %s is not approved", ticket.ID), RiskReport: report}, nil
        }
        if len(ticket.Approvers) < profile.RequireApprovals {
            return &Decision{Allowed: false, Code: "INSUFFICIENT_APPROVALS", Message: fmt.Sprintf("delete denied: require %d approvals, got %d", profile.RequireApprovals, len(ticket.Approvers)), RiskReport: report}, nil
        }
    }
    if report.Level == RiskCritical {
        return &Decision{Allowed: true, Code: "ALLOW_WITH_COOLDOWN", Message: fmt.Sprintf("delete accepted with cooldown: risk=%s score=%d", report.Level, report.Score), RiskReport: report}, nil
    }
    return &Decision{Allowed: true, Code: "ALLOW", Message: fmt.Sprintf("delete allowed: risk=%s score=%d", report.Level, report.Score), RiskReport: report}, nil
}

Production‑Ready Go Server

The article provides a minimal HTTP server that decodes AdmissionReview objects, invokes the validator, and writes the response. It logs request details for observability and respects the failurePolicy semantics.

func (s *Server) handleValidate(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    body, err := io.ReadAll(r.Body)
    if err != nil { http.Error(w, "read body failed", http.StatusBadRequest); return }
    review := &admissionv1.AdmissionReview{}
    if _, _, err := deserializer.Decode(body, nil, review); err != nil { http.Error(w, "decode admission review failed", http.StatusBadRequest); return }
    req := review.Request
    resp := &admissionv1.AdmissionResponse{UID: req.UID}
    if req.Operation != admissionv1.Delete { resp.Allowed = true; s.writeReview(w, review, resp); return }
    oldMeta := &metav1.PartialObjectMetadata{}
    if len(req.OldObject.Raw) > 0 { json.Unmarshal(req.OldObject.Raw, oldMeta) }
    decision, err := s.validator.ValidateDelete(r.Context(), req, oldMeta)
    if err != nil { resp.Allowed = false; resp.Result = &metav1.Status{Message: "internal validation error"}; s.writeReview(w, review, resp); return }
    resp.Allowed = decision.Allowed
    resp.Result = &metav1.Status{Message: decision.Message, Reason: metav1.StatusReason(decision.Code)}
    klog.Infof("delete validation kind=%s ns=%s name=%s allowed=%v code=%s cost=%s", req.Kind.Kind, req.Namespace, req.Name, resp.Allowed, decision.Code, time.Since(start))
    s.writeReview(w, review, resp)
}

Dependency Analysis Engine

Because full‑cluster scans are too expensive, the design recommends an informer‑based indexer that maintains in‑memory reverse indexes (namespace → resources, ownerUID → children, service → ingress). The webhook reads from this cache, and a short‑TTL hotspot cache avoids repeated work for hot resources.

Buffer Layer with Finalizer

For high‑risk deletions the system adds a custom finalizer (e.g., sre.example.com/deletion-guard) so the API server marks the object as Terminating. A controller watches the finalizer, records audit snapshots, notifies external systems, waits for a configurable cool‑down (e.g., 10 minutes), optionally requires human confirmation, and finally removes the finalizer to complete the delete.

func (r *FinalizerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    obj := &unstructured.Unstructured{}
    obj.SetAPIVersion("apps/v1")
    obj.SetKind("Deployment")
    if err := r.Get(ctx, req.NamespacedName, obj); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) }
    if obj.GetDeletionTimestamp().IsZero() { return ctrl.Result{}, nil }
    if !containsFinalizer(obj.GetFinalizers(), DeletionGuardFinalizer) { return ctrl.Result{}, nil }
    if time.Since(obj.GetDeletionTimestamp().Time) < r.CoolDown { return ctrl.Result{RequeueAfter: 30 * time.Second}, nil }
    obj.SetFinalizers(removeFinalizer(obj.GetFinalizers(), DeletionGuardFinalizer))
    if err := r.Update(ctx, obj); err != nil { return ctrl.Result{}, err }
    return ctrl.Result{}, nil
}

High‑Availability and Scaling

Key recommendations:

Run the webhook as at least three replicas with PodAntiAffinity, PDB, HPA, and readiness probes.

Exclude system namespaces (kube‑system, gatekeeper‑system, cert‑manager, sre‑system) from protection to avoid self‑lock.

Keep the synchronous path under 500 ms – 2 s; set timeoutSeconds to 2‑3 s.

Cache results per resource with a 10‑30 s TTL; de‑duplicate identical requests.

Apply resource‑type‑specific rate limits (e.g., stricter limits for Namespace/CRD deletions).

Observability and Auditing

Every decision is emitted as a structured audit event (JSON example omitted for brevity) and Prometheus metrics are exposed:

var (
    deletionRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{Name: "deletion_guard_requests_total", Help: "Total number of delete validation requests"}, []string{"kind", "decision", "code"})
    deletionValidationLatency = promauto.NewHistogramVec(prometheus.HistogramOpts{Name: "deletion_guard_validation_seconds", Help: "Latency of delete validation", Buckets: []float64{0.01,0.03,0.05,0.1,0.2,0.5,1,2}}, []string{"kind"})
    dependencyCacheHitTotal = promauto.NewCounterVec(prometheus.CounterOpts{Name: "deletion_guard_dependency_cache_total", Help: "Dependency cache hit and miss counts"}, []string{"result"})
)

Dashboards track request pass/fail rates, P99 latency per resource type, top denial reasons, and finalizer‑stuck objects. Alerts fire on high webhook error rates, latency spikes, or abnormal finalizer counts.

Real‑World Scenarios

Three production cases illustrate the system:

Accidental deletion of a critical Deployment: The resource carries sre/protection-level=critical. Validation scores 95 (CRITICAL) and, lacking a ticket, the webhook denies the request.

Planned removal of a StatefulSet with approval: Ticket passes, risk is CRITICAL, the request is accepted, a finalizer adds a 10‑minute cool‑down, notifications are sent, and the object is finally deleted after the window.

CI batch cleanup of test Deployments: Authorization quickly allows non‑critical resources, validation hits a cache, and the webhook stays under the latency budget even for 200 simultaneous deletes.

Rollout Strategy

A four‑phase rollout (Observe‑Only → Fail‑Closed in dev → Partial protect in production → Full protect) ensures safe adoption. During Observe‑Only the validation ring records what would have been denied without blocking traffic.

Common Pitfalls and Mitigations

Webhook self‑lock: Exclude the protection system’s own namespace and ServiceAccount.

Finalizer residue: Implement timeout removal and monitor finalizer age.

Slow analysis for large resources: Use resource‑type‑specific strategies, caching, and async enrichment.

Insufficient label coverage: Enforce label injection in CI templates and run periodic audits.

Bypass abuse: Require audit logging and ticket linkage for every bypass.

Comparison with Alternative Approaches

Pure RBAC lacks contextual checks; pure Kyverno/YAML policies are great for simple rules but cannot handle external approvals or dynamic risk scoring. The presented OPA + custom webhook approach balances flexibility, performance, and auditability.

Future Evolution

After stabilizing the double‑ring, the roadmap includes automatic protection‑level recommendation based on historical usage, anomaly detection for unusual delete patterns, and predictive risk modeling that incorporates live traffic and SLO data.

Implementation Checklist

Define protection‑level model and label schema.

Identify critical resource types and namespaces.

Deploy an Observe‑Only webhook for 2‑4 weeks to collect data.

Implement the Authorization Ring via OPA Gatekeeper.

Add the Validation Ring with risk scoring and ticket integration.

Enable Finalizer‑based buffering for high‑risk objects.

Configure multi‑replica webhook, readiness probes, and PDB.

Instrument Prometheus metrics and audit publishing.

Run failure‑mode drills (webhook outage, cache miss, finalizer stuck).

Gradually expand protection scope following the phased rollout.

Conclusion

Effective Kubernetes deletion protection requires separating "can delete" from "should delete", adding a buffering stage, and closing the loop with audit and observability. The dual‑ring architecture delivers a programmable, high‑availability, and extensible solution that turns deletion governance from a manual safeguard into a systematic, measurable capability.