K8s Multi‑Tenant Isolation: Practical Hierarchical Namespaces with Namespace + HNC

Define isolation boundaries before building a hierarchy

Multi‑tenant design starts with clearly writing the isolation boundaries rather than merely drawing a namespace tree. Different tenants (departments, external customers, environments) have varying risk levels, requiring distinct controls for workload creation, CPU/memory limits, service access, secret reading, privileged pods, policy propagation, and lifecycle management.

Name & Objects : Prevent workload, Service, ConfigMap name collisions – achieved with Namespace naming and hierarchical organization; still need admission policies and GitOps paths.

Permissions : Control API rights for users and ServiceAccounts – use namespace‑scoped Role/RoleBinding with inheritance; still need RBAC audit, least‑privilege, and temporary grants.

Resources : Limit CPU, memory, Pods, PVCs, object counts – enforce with ResourceQuota and LimitRange per sub‑namespace; node pools, PriorityClass, and cost accounting remain external.

Network : Control Pod‑to‑Pod, Pod‑to‑external, DNS, control‑plane access – implement with NetworkPolicy per namespace; requires CNI capabilities, egress gateways, and L7 policies.

Runtime : Restrict privileged, root, hostPath, capabilities – trigger Pod Security Admission via namespace labels; still need image admission, RuntimeClass, and audit.

Lifecycle : Manage creation, migration, freeze, recycle, and recovery – use HNC parent‑child relations and anchor lifecycle; still need backup, change approval, and data retention policies.

Namespace, RBAC, and NetworkPolicy provide logical isolation but share the same API server, node kernel, CNI, and storage. For untrusted code or strict compliance, consider separate clusters, node pools, sandbox runtimes, or virtual clusters.

Pre‑install compatibility and permission audit

HNC works via a CRD and controller that watches Namespaces, hierarchical objects, and propagated resources. Before installing, verify version compatibility, controller RBAC, admission webhooks, existing GitOps controllers, and resource propagation rules.

# Verify cluster version, HNC CRD and controller objects
kubectl -n <platform-namespace> version --client --output=yaml
kubectl api-resources -n <platform-namespace> | rg -i 'hnc|hierarchy|subnamespace'
kubectl get crd -n <platform-namespace> | rg 'hnc.x-k8s.io'
kubectl get deployment -n <platform-namespace> -o wide

If the HNC types are missing, do not apply the YAML directly; let the platform team install the approved version and validate it against the cluster’s Kubernetes version, Pod Security Admission, Gatekeeper/Kyverno, GitOps controller, and CNI.

# Export baseline resources for review
tenant_root="<tenant-root-namespace>"
platform_namespace="<platform-namespace>"
backup_dir="<backup-dir>"
mkdir -p "$backup_dir"
kubectl get resourcequota -n "$tenant_root" -o yaml > "$backup_dir/quota.before.yaml"
kubectl get limitrange -n "$tenant_root" -o yaml > "$backup_dir/limitrange.before.yaml"
kubectl get networkpolicy -n "$tenant_root" -o yaml > "$backup_dir/networkpolicy.before.yaml"
kubectl get hncconfigurations -n "$platform_namespace" -o yaml > "$backup_dir/hnc-config.before.yaml"

Design a governable hierarchy

Keep the hierarchy shallow – at most three levels: tenant‑root, environment (dev/test/prod), and project/service. Deeper trees hard‑code organizational structure and increase maintenance cost.

# Tenant‑root namespace security baseline (Pod Security Admission)
apiVersion: v1
kind: Namespace
metadata:
  name: <tenant-root-namespace>
  labels:
    tenant.example.com/name: <tenant-id>
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn-version: v1.27
    pod-security.kubernetes.io/audit-version: v1.27

Apply the manifest with a server‑side dry‑run to catch admission‑time errors before real creation.

# Dry‑run and apply tenant‑root namespace
kubectl apply -n <tenant-root-namespace> --dry-run=server -f tenant-root-namespace.yaml
kubectl diff -n <tenant-root-namespace> -f tenant-root-namespace.yaml || true
kubectl apply -n <tenant-root-namespace> -f tenant-root-namespace.yaml
kubectl get resourcequota -n <tenant-root-namespace>

Create sub‑namespaces with SubnamespaceAnchor

HNC creates a child namespace when a SubnamespaceAnchor object is applied in the parent. Deleting the anchor may affect the child hierarchy, so test in a non‑critical tenant first.

# Create a sub‑namespace anchor
apiVersion: hnc.x-k8s.io/v1alpha2
kind: SubnamespaceAnchor
metadata:
  name: <child-namespace>
  namespace: <tenant-root-namespace>
  labels:
    tenant.example.com/environment: dev

# Verify anchor and hierarchy status
kubectl diff -n <tenant-root-namespace> -f dev-anchor.yaml || true
kubectl apply -n <tenant-root-namespace> -f dev-anchor.yaml
kubectl get subnamespaceanchor -n <tenant-root-namespace> <child-namespace> -o yaml
kubectl get hierarchyconfiguration -n <child-namespace> hierarchy -o yaml
kubectl get events -n <tenant-root-namespace> --sort-by=.lastTimestamp

Quota and default resources: "can create" ≠ "can exhaust"

ResourceQuota caps total consumption, while LimitRange sets defaults and per‑object bounds. Both must be defined in each sub‑namespace because a quota on the root does not automatically apply downstream.

# Example ResourceQuota for a development sub‑namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-budget
  namespace: <child-namespace>
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "30"
    services: "20"
    persistentvolumeclaims: "10"

# LimitRange providing default requests and limits
apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: <child-namespace>
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    default:
      cpu: "1"
      memory: 1Gi
    min:
      cpu: 50m
      memory: 64Mi
    max:
      cpu: "2"
      memory: 4Gi

After applying, inspect kubectl describe resourcequota and kubectl describe limitrange to ensure the hard limits and defaults match expectations.

Network isolation: default deny then allow as needed

Without a NetworkPolicy, most CNI plugins allow unrestricted pod‑to‑pod traffic. Apply a default‑deny policy in each sub‑namespace, then explicitly allow DNS, ingress controller, and required services.

# Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: <child-namespace>
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# Allow DNS from kube-system
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: <child-namespace>
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Validate the policy by checking that pods can resolve DNS but cannot reach other namespaces unless explicitly allowed. Use a diagnostic pod (no hostPath, no privileged caps) to test connectivity.

# Diagnostic pod for network checks
apiVersion: v1
kind: Pod
metadata:
  name: netcheck
  namespace: <child-namespace>
  labels:
    app: netcheck
spec:
  restartPolicy: Never
  containers:
  - name: netcheck
    image: <image>@sha256:<digest>
    command: ["sh", "-c", "sleep 3600"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

RBAC: parent convenience must not become child over‑privilege

HNC can propagate RoleBindings, which may unintentionally grant higher privileges in child namespaces. Define minimal roles per application and bind them only to the appropriate group.

# Minimal deployment Role in a child namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workload-deployer
  namespace: <child-namespace>
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get","list","watch","create","update","patch"]
- apiGroups: [""]
  resources: ["services","configmaps"]
  verbs: ["get","list","watch","create","update","patch"]
- apiGroups: [""]
  resources: ["pods","pods/log"]
  verbs: ["get","list","watch"]

# Bind the minimal role to a team group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workload-deployer-binding
  namespace: <child-namespace>
subjects:
- kind: Group
  name: <team-group>
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: workload-deployer
  apiGroup: rbac.authorization.k8s.io

Test the binding with kubectl auth can-i as the target user/group, checking both allowed and denied actions (e.g., secret read, rolebinding creation).

Propagation strategy, conflict handling, and observability

Define a clear propagation policy for each resource type: source namespace, inheritance scope, conflict behavior, owner, rollback method, and validation metrics. Use read‑only commands to verify that propagated objects have the expected labels, selectors, and permissions.

# Verify propagated objects in a child namespace
kubectl get hierarchyconfiguration -n <child-namespace> hierarchy -o yaml
kubectl get resourcequota -n <child-namespace> -o yaml
kubectl get limitrange -n <child-namespace> -o yaml
kubectl get networkpolicy -n <child-namespace> -o yaml
kubectl get rolebinding -n <child-namespace> -o yaml

Monitor quota usage with Prometheus (example query shown) and treat sustained near‑limit usage combined with scheduling failures as a signal to increase capacity or clean up.

# Example Prometheus query for quota usage ratio
max by (namespace, resource) (kube_resourcequota{namespace="<child-namespace>",type="used"}) /
max by (namespace, resource) (kube_resourcequota{namespace="<child-namespace>",type="hard"})

High‑risk operations: move, delete, and rollback

Deleting an anchor, enabling cascading delete, changing a parent, or propagating a high‑privilege RoleBinding can affect all descendant workloads. Before such actions, list the impact tree, backup resources, and obtain tenant approval.

# Read‑only impact assessment before deletion or move
kubectl get subnamespaceanchor -n <tenant-root-namespace> <child-namespace> -o yaml
kubectl get pods -n <child-namespace> -o wide
kubectl get pvc -n <child-namespace>
kubectl get ingress -n <child-namespace>
kubectl get events -n <child-namespace> --sort-by=.lastTimestamp

Rollback by re‑applying the previously exported YAML files and verifying that the system returns to the expected state.

# Rollback resources from backup
namespace="<child-namespace>"
backup_dir="<backup-dir>"
for file in "$backup_dir/quota.before.yaml" "$backup_dir/limitrange.before.yaml" "$backup_dir/networkpolicy.before.yaml"; do
  test -r "$file"
  kubectl diff -n "$namespace" -f "$file" || true
  kubectl apply -n "$namespace" -f "$file"
done
kubectl get events -n "$namespace" --sort-by=.lastTimestamp

A sustainable multi‑tenant solution treats HNC as the mechanism to express hierarchy, while Namespace, RBAC, NetworkPolicy, and quotas provide the actual isolation. Each node in the tree must have an owner, limited permissions, measurable resources, default network boundaries, observable state, and a rehearsed reclamation path.