Deep Hardening of Kubernetes Production Clusters: From Running to Sleeping Soundly

Why Many Clusters Appear Stable Yet Remain High‑Risk

Teams often mistake a running pod, healthy service connectivity, low CPU usage, and scanned images for a secure cluster, but real production incidents follow a chain: missing network policies, privileged pods, lateral scans, conntrack table exhaustion, kube‑proxy jitter, API‑server watch pressure, amplified thread‑pool retries, and finally 504 errors or cascading outages.

Seven‑Layer Defense Model

The article proposes a seven‑layer view that converges risk from the supply‑chain layer up to the observation layer, each layer addressing a specific attack surface:

Supply‑chain: image build, signing, SBOM, scanning.

Admission: PSA, OPA/Gatekeeper, namespace baselines.

Network: default‑deny, L7 policies, host firewall.

Runtime: seccomp, AppArmor/SELinux, Falco/Tetragon, sandbox runtimes.

Control plane: etcd, API server, kubelet, kernel parameters.

Workloads: HPA/VPA/KEDA, PDB, priority, topology spread.

Observability & response: metrics, audit logs, alerts.

Defining Goals and SLOs

Four categories of objectives are recommended:

Security baseline : enforce PSA baseline or restricted, block unsigned images, disallow :latest tags, require non‑root.

Availability : tolerate a single node or AZ failure, keep API‑server writes alive during restarts.

Performance : monitor API‑server P99 latency, etcd fsync latency, CoreDNS response time, conntrack usage.

Engineering : GitOps for policies, CI‑CD security scans, operator‑driven middleware, standardized emergency drills.

1. Admission Security – From PSA to Gatekeeper

PodSecurityAdmission (PSA) is the first gate. Use baseline for most workloads and upgrade to restricted for high‑risk namespaces. PSA cannot express complex conditions, so layer OPA/Gatekeeper policies on top.

Namespace layering example (production namespace):

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:
      cpu: "1"
      memory: 1Gi
    defaultRequest:
      cpu: 200m
      memory: 256Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
    limits.cpu: "200"
    limits.memory: 400Gi
    pods: "500"
    persistentvolumeclaims: "100"

Key outcomes of the above YAML:

Prevents pods without resource requests from stealing nodes.

Stops a single team from exhausting cluster capacity.

Combines PSA with explicit resource quotas for a double‑door security model.

Production‑grade Deployment example (order‑service):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 4
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
        tier: backend
    spec:
      serviceAccountName: order-service
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: app
        image: registry.example.com/order-service:v2.3.7
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75"
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: "2"
            memory: 2Gi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsUser: 10001
          runAsGroup: 10001
          capabilities:
            drop: ["ALL"]
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        startupProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          failureThreshold: 30
          periodSeconds: 5
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: logs
          mountPath: /var/log/app
      volumes:
      - name: tmp
        emptyDir: {}
      - name: logs
        emptyDir: {}

Four production‑grade details highlighted:

Disable automatic ServiceAccount token mounting.

Enforce read‑only root filesystem with explicit writable paths.

Separate startup from liveness probes to avoid premature restarts of cold‑start services.

Declare both resource requests and limits so HPA/VPA have accurate data.

Gatekeeper policy examples (disallow :latest, enforce resource requests, require owner/cost‑center/env labels, whitelist image registries, block host‑network, host‑PID, host‑path):

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowlatest
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowLatest
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8sdisallowlatest
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        endswith(container.image, ":latest")
        msg := sprintf("container %v uses disallowed latest tag", [container.name])
      }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowLatest
metadata:
  name: disallow-latest-tag
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]

2. Supply‑Chain Security

Key steps to keep untrusted images out of the cluster:

Image minimization – use distroless, Alpine, or an organization‑wide base image.

Vulnerability scanning – Trivy, Grype, or cloud‑provider scanners.

Generate SBOM for traceability.

Sign images with Cosign and verify signatures at admission.

Typical CI flow:

# Build image & generate SBOM
# Sign image with Cosign
# Push to internal registry
# Admission layer only accepts images from the internal registry with a valid signature

3. Network Security – Micro‑Segmentation

Kubernetes defaults to a flat network. Production must switch to a whitelist model.

Default‑deny policies (ingress & egress):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress

Whitelist example allowing only frontend and DNS for the order-service pod:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: order-allow-frontend-and-dns
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: order-service
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: middleware
      podSelector:
        matchLabels:
          app: kafka
    ports:
    - protocol: TCP
      port: 9093

Native NetworkPolicy works for small clusters but lacks L7, high‑performance data plane, and identity‑based policies. When those are needed, the article recommends moving to Cilium, which provides L7 rules, better observability, and host‑firewall integration.

Cilium L7 egress example for a payment API:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-egress-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: order-service
  egress:
  - toFQDNs:
    - matchName: payment-api.internal.example.com
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/api/v1/payments"
        - method: "GET"
          path: "/api/v1/payments/.*"

Host firewall example to lock down SSH on worker nodes:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: restrict-node-ssh
spec:
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  ingress:
  - fromCIDR:
    - 10.10.0.0/16
    toPorts:
    - ports:
      - port: "22"
        protocol: TCP

4. Runtime Security

Before deploying Falco, lock down the kernel surface:

Set seccompProfile: RuntimeDefault on all pods.

Enable AppArmor or SELinux on nodes.

For high‑risk workloads, apply custom seccomp/AppArmor profiles.

Falco rule example detecting an interactive shell inside a container:

- rule: Terminal shell in container
  desc: A shell was spawned in a container
  condition: container and proc.name in (bash, sh, zsh, ash)
  output: Shell started in container user=%user.name container=%container.name pod=%k8s.pod.name namespace=%k8s.ns.name
  priority: WARNING
  tags: [container, shell]

Tetragon tracing policy to catch package‑manager binaries (apt, yum) execution:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-package-manager
spec:
  kprobes:
  - call: "execve"
    syscall: true
    args:
    - index: 0
      type: "string"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "/usr/bin/apt"
        - "/usr/bin/apt-get"
        - "/usr/bin/yum"

Sandbox runtimes for untrusted code: gVisor – user‑space kernel interception, moderate overhead. Kata Containers – lightweight VM isolation, higher overhead.

RuntimeClass example for Kata:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata
---
apiVersion: v1
kind: Pod
metadata:
  name: tenant-job-runner
  namespace: sandbox
spec:
  runtimeClassName: kata
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: job
    image: registry.example.com/tenant-runner:v1.0.0
    securityContext:
      allowPrivilegeEscalation: false
      runAsUser: 65532

5. Control‑Plane Stability

etcd – use dedicated SSDs, avoid co‑locating high‑IO pods, run periodic compact and defrag, monitor backend size, fsync duration, and leader changes. Example etcd container args:

etcd \
  --quota-backend-bytes=8589934592 \
  --heartbeat-interval=250 \
  --election-timeout=5000 \
  --snapshot-count=10000 \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1

API Server – beyond --max-requests-inflight, enable API Priority and Fairness (APF) to protect system‑level controllers from noisy‑batch workloads.

kubelet & containerd – tune for high concurrency: increase maxPods, disable serializeImagePulls, set aggressive image GC thresholds, configure eviction hard limits, and set topologyManagerPolicy: best-effort.

# /var/lib/kubelet/config.yaml
maxPods: 200
serializeImagePulls: false
imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 60
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
topologyManagerPolicy: best-effort

Kernel parameters – tune for large clusters:

# /etc/sysctl.d/99-kubernetes.conf
net.core.somaxconn = 32768
net.ipv4.ip_local_port_range = 1024 65000
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
fs.file-max = 1048576
fs.inotify.max_user_watches = 524288
vm.swappiness = 0
vm.overcommit_memory = 1

Key metrics to watch: nf_conntrack usage, available local ports, disk latency.

6. Workload Engineering – Autoscaling, Disruption Budgets, Priority

Clarify responsibilities:

HPA – horizontal pod scaling based on CPU, memory, QPS, etc.

VPA – recommend resource requests for a pod.

KEDA – event‑driven scaling (e.g., Kafka lag).

Production‑grade HPA for order-service (fast scale‑up, slow scale‑down):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 4
  maxReplicas: 30
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
      - type: Pods
        value: 4
        periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 20
        periodSeconds: 60
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65

KEDA ScaledObject for a Kafka consumer (lag‑driven scaling):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-consumer
  namespace: production
spec:
  scaleTargetRef:
    name: order-consumer
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 3
  maxReplicaCount: 50
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: my-kafka-kafka-bootstrap.middleware:9092
      consumerGroup: order-consumer-group
      topic: order-events
      lagThreshold: "5000"

PodDisruptionBudget, PriorityClass, and topology spread constraints to protect against node maintenance and AZ failures:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-service-pdb
  namespace: production
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: order-service
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: core-online-service
value: 100000
globalDefault: false
description: "Priority for core online services"

Deployment topology spread:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: order-service

7. Data‑Plane Middleware Production‑Grade

Prefer operators over raw StatefulSets to encode operational knowledge. Kafka on Kubernetes – Strimzi example with dual‑role node pool and storage configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dual-role
  labels:
    strimzi.io/cluster: my-kafka
spec:
  replicas: 3
  roles:
  - controller
  - broker
  storage:
    type: jbod
    volumes:
    - id: 0
      type: persistent-claim
      size: 500Gi
      class: ssd-storage-class
      deleteClaim: false
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka
  namespace: middleware
spec:
  kafka:
    version: 4.0.0
    replicas: 3
    listeners:
    - name: tls
      port: 9093
      type: internal
      tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      num.partitions: 24
      log.retention.hours: 168
      log.segment.bytes: 1073741824
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: strimzi.io/name
                  operator: In
                  values:
                  - my-kafka-kafka
              topologyKey: topology.kubernetes.io/zone

Capacity estimation for a high‑throughput service (order‑service) follows four steps: measure per‑pod steady‑state QPS, compute required replicas (peak QPS ÷ pod QPS × safety factor), ensure node capacity for summed requests (reserve 15‑30% headroom), and verify downstream capacity (DB connections, Kafka partitions, cache hit rate).

8. Observability & Incident Response

Five metric groups to monitor:

Control plane – API Server latency/error, etcd fsync, leader changes.

Network – conntrack usage, DNS latency, packet loss.

Workloads – HPA behavior, pod restarts, P99 latency, queue backlog.

Security – PSA deny count, Gatekeeper violations, Falco/Tetragon events.

Middleware – Kafka lag, ISR changes, Nacos connections, MySQL response time.

Audit log must capture create/update/patch/delete, exec/port‑forward/attach, RBAC changes, and secret reads. A Bash script is provided to list namespaces with PSA labels, privileged pods, missing NetworkPolicies, and node conntrack usage.

#!/usr/bin/env bash
set -euo pipefail

echo "=== Namespaces with PSA labels ==="
kubectl get ns -o json | jq -r '.items[] | [.metadata.name, (.metadata.labels["pod-security.kubernetes.io/enforce"] // "none"), (.metadata.labels["pod-security.kubernetes.io/warn"] // "none")] | @tsv'

# ... (script continues as in the article) ...

Alert design should combine symptom metrics (gateway 5xx, order‑service P99, Kafka lag) with root‑cause indicators (conntrack saturation, CoreDNS latency) to quickly pinpoint whether the issue is upstream load or underlying network/node failure.

9. Real‑World Case Study – Order Platform Transformation

The article walks through a concrete migration from a fragile state to a stable one:

Initial problems: missing PSA/NetworkPolicy, root containers, CPU‑only HPA, Kafka sharing nodes, no audit log retention.

Incident during load test: gateway 504s, order‑service RT spikes, Kafka lag explosion, conntrack near limit, CoreDNS slowdown.

Phase‑1 actions – enable baseline PSA, default‑deny policies, whitelist network rules.

Phase‑2 actions – RuntimeDefault seccomp, Falco audit mode, raise conntrack limits, move etcd to SSD, add API Server & CoreDNS alerts.

Phase‑3 actions – KEDA‑driven consumer scaling, migrate Kafka to Strimzi on dedicated nodes, add PDB/priority/topology spread, enforce image signature checks.

Result – reduced gateway errors, smoother scaling, no cascade failures, security policies automated via GitOps, faster MTTR.

10. Phased Rollout Plan

Phase 1 – Baseline visibility (1‑2 weeks) : inventory namespaces, enable PSA warn/audit, apply default‑deny NetworkPolicy template, ingest control‑plane and node metrics.

Phase 2 – Basic closure (2‑4 weeks) : enforce PSA baseline/restricted per namespace, add resource request policies, implement whitelist network rules for core services, add PDB, PriorityClass, topology spread.

Phase 3 – Deep defense (4‑8 weeks) : roll out Falco/Tetragon, sandbox high‑risk workloads with gVisor/Kata, enforce image signing at admission, harden Kafka/Nacos with operators and dedicated nodes.

Phase 4 – Continuous operation (ongoing) : manage all policies via GitOps, integrate security baseline checks into CI/CD, run regular capacity and chaos tests, review alert effectiveness and resource profiles.

11. Checklist – How Far Is Your Cluster From "Sleeping Soundly"?

Do new namespaces automatically receive PSA, quota, LimitRange, and default‑deny policies?

Can you enumerate pods still running as root or privileged?

Is any outbound traffic unrestricted?

Can you locate the source of an exec inside a container, RBAC change, or secret read within five minutes?

Do you know the safe water‑mark for nf_conntrack on each node?

Are etcd fsync latency and fragmentation monitored?

Do core services have PDB, topology spread, and priority class configured?

Are stateful middlewares (Kafka, Nacos) isolated on dedicated nodes with fault‑domain awareness?

Does the CI/CD pipeline verify image provenance, signatures, and version traceability?

Are all security policies version‑controlled, auditable, and rollback‑capable?

12. Final Takeaway

Turning a Kubernetes cluster from merely runnable into reliably stable is not about flipping a few switches; it is about systematically managing uncertainty across admission, network, runtime, control‑plane, workload, and observability layers. When each layer is hardened, the cluster can absorb traffic spikes, node failures, and malicious activity while keeping the business running and the team sleeping peacefully.