From CI Breach to Sustainable Governance: Kubernetes Access Control End-to-End Guide

0. Real Incident Model: How a Temporary cluster-admin Became a Cluster-Wide Incident

A release agent ran in the ci namespace. During troubleshooting, an engineer bound its ServiceAccount to cluster-admin and never revoked it. Later, a poisoned build dependency gave attackers command execution inside the agent container.

The attack chain used legitimate API calls, not container escape:

CI Agent RCE
 → Read ServiceAccount credentials from Pod
 → Call kube-apiserver
 → Create ClusterRoleBinding / privileged workloads
 → Read cross-namespace Secrets, access nodes, lateral movement

The lesson: limit what a compromised identity can do via identity trust + least RBAC + admission constraints + network isolation + audit detection . No single layer replaces another.

1. Establish Correct Boundaries: What Every Request Traverses

All kubectl, controllers, operators, and CI/CD operations pass through kube-apiserver:

TLS
 → Authentication (who you are)
 → Authorization (can you perform this verb)
 → Mutating Admission (modify object?)
 → Object Validation / Validating Admission (allow object?)
 → Persist to etcd

Authentication establishes username, groups, ServiceAccount identity; grants no permissions.

Authorization decides if an identity can get, patch, create, etc. on a resource; RBAC is the common implementation.

Admission restricts "what a permitted creator actually creates." Example: having create pods must not allow privileged pods.

Therefore RBAC cannot replace PSA, admission policies, or NetworkPolicy; NetworkPolicy does not limit Kubernetes API permissions.

2. Identity & Credentials: Humans, CI, and Workload Pods Must Be Separated

2.1 Human Users: Enterprise IdP → Groups → RBAC

Production must forbid shared admin.conf or shared kubeconfig. Use enterprise IdP (OIDC/JWT or cloud-native identity integration) so API server sees auditable personal identities and groups:

[email protected]
groups: [oidc:team-order, oidc:sre]

Bind RBAC to groups, not individuals:

subjects:
- kind: Group
  name: oidc:team-order
  apiGroup: rbac.authorization.k8s.io

Reject IdP mapping external users to system: prefix (Kubernetes reserved). Employee join/move/leave handled by IdP; Kubernetes only consumes group info.

2.2 Workload Pods: Default to No API Credentials

Pods that don't need Kubernetes API access should disable auto-mounted ServiceAccount tokens:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-runtime
  namespace: team-order
automountServiceAccountToken: false

Pod-level automountServiceAccountToken: false overrides ServiceAccount setting. Since v1.22, projected tokens (short-lived, auto-rotated via TokenRequest) replace static Secret tokens. Short-lived credentials shrink exposure window but don't reduce cluster-admin risk; high-privilege identity can establish persistence within an hour.

2.3 CI/CD: Disabling Default Mount ≠ No Credentials

If CI Pod calls kube-apiserver, setting automountServiceAccountToken: false alone breaks deployments. Explicitly project a short-lived token; application reads from file and supports rotation (client-go supports bearer token file):

apiVersion: v1
kind: Pod
metadata:
  name: deploy-job-example
  namespace: ci
spec:
  serviceAccountName: deploy-agent
  automountServiceAccountToken: false
  containers:
  - name: deployer
    image: registry.example.com/deployer:1.0.0
    volumeMounts:
    - name: kube-api-token
      mountPath: /var/run/tokens
      readOnly: true
  volumes:
  - name: kube-api-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          audience: https://kubernetes.default.svc
          expirationSeconds: 3600

For external cloud CI, prefer cloud workload identity or OIDC federation for short-lived cluster credentials; avoid storing long-lived ServiceAccount tokens in CI variables. Manually created kubernetes.io/service-account-token Secrets only for explicit compatibility, with recorded owner, purpose, and rotation schedule.

3. RBAC: Draw Permission Boundaries First, Then Write YAML

Recommended permission matrix by identity:

Developers : Observe, limited workload changes in dev/test namespaces. Explicitly denied: production writes, Secrets, RBAC, pods/exec.

Release Agent : Deploy required objects in target namespace. Explicitly denied: Secrets, RBAC, cross-namespace, nodes, privileged debug.

SRE : Observe & controlled ops. Explicitly denied: default permanent cluster-admin.

Security Audit : Metadata, audit systems. Explicitly denied: Secret values, workload writes.

Break Glass : Approved short-lived high privilege. Explicitly denied: standing authorization. ClusterRole + RoleBinding is the recommended pattern for reusable templates per namespace. Note: RoleBinding referencing a ClusterRole applies namespace-scoped permissions only within that RoleBinding's namespace; ClusterRoleBinding makes it cluster-wide.

3.1 Production Release Role: Can Deploy, Cannot Touch Permission Boundaries

Example Role for team-order GitOps/direct deploy. Allows create/update/delete of deployments, services, configmaps, poddisruptionbudgets. Intentionally excludes secrets, serviceaccounts, pods/exec, roles, rolebindings, clusterroles, clusterrolebindings, nodes. Add Ingress, HPA, or specific CRDs individually; never use resources: ["*"].

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: order-deployer
  namespace: team-order
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "deployments/scale"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Cross-namespace binding for CI identity:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-order-deployer
  namespace: team-order
subjects:
- kind: ServiceAccount
  name: deploy-agent
  namespace: ci
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: order-deployer

3.2 Workload Creation Permission Is Itself High-Risk

Not granting create pods directly isn't enough. Identities with Deployment, StatefulSet, Job, or CronJob create rights can indirectly create pods and specify any ServiceAccount in the same namespace.

Must enforce three controls together:

Business namespaces must not host high-privilege ServiceAccounts; controllers/platform components go to dedicated namespaces.

Create dedicated, least-privilege order-runtime ServiceAccount for runtime.

Use admission policies to restrict which ServiceAccount production workloads may use, and PSA to restrict dangerous pod shapes.

Missing any one lets the release identity escalate via workload templates.

3.3 Real Limits of RBAC Write Permissions

Kubernetes API blocks ordinary users from gaining new permissions by creating Role/RoleBinding; escalate and bind are high-risk exceptions requiring focused audit. Remaining escalation paths to review:

bind / escalate / impersonate
Create/modify Role, ClusterRole, RoleBinding, ClusterRoleBinding
Read Secrets
Create workloads selecting high-privilege ServiceAccount
Patch ServiceAccount
pods/exec, pods/attach, pods/portforward

4. Admission: PSA as Baseline, Policy Engine for Organizational Rules

4.1 Reliable PSA Rollout Sequence

First add warn and audit labels to production namespaces, observe and remediate non-compliant workloads; only after verification add enforce. Never enforce first then migrate.

kubectl label namespace team-order \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn-version=v1.30 \
  pod-security.kubernetes.io/audit-version=v1.30

# After remediation & rollback drills
kubectl label namespace team-order \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.30

Replace version with actual control-plane compatible version; include impact of raising policy standard after upgrades in drills; avoid long-term latest in production. PSA provides privileged, baseline, restricted tiers but cannot express organizational rules like "this workload must use a specific ServiceAccount."

4.2 Constrain Production Pods with ValidatingAdmissionPolicy

Policy binds only to namespaces labeled security.example.com/tier=production. Denies privileged containers, host namespaces, hostPath, and forces order-runtime ServiceAccount. Checks regular, init, and ephemeral containers.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: production-pod-baseline
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: "object.spec.serviceAccountName == 'order-runtime'"
    message: "production Pods must use ServiceAccount order-runtime"
  - expression: "!has(object.spec.hostNetwork) || object.spec.hostNetwork == false"
    message: "hostNetwork is not allowed"
  - expression: "!has(object.spec.hostPID) || object.spec.hostPID == false"
    message: "hostPID is not allowed"
  - expression: "!has(object.spec.hostIPC) || object.spec.hostIPC == false"
    message: "hostIPC is not allowed"
  - expression: "!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath))"
    message: "hostPath volumes are not allowed"
  - expression: "object.spec.containers.all(c, !has(c.securityContext) || ((!has(c.securityContext.privileged) || c.securityContext.privileged == false) && (!has(c.securityContext.allowPrivilegeEscalation) || c.securityContext.allowPrivilegeEscalation == false)))"
    message: "containers may not be privileged or allow privilege escalation"
  - expression: "!has(object.spec.initContainers) || object.spec.initContainers.all(c, !has(c.securityContext) || ((!has(c.securityContext.privileged) || c.securityContext.privileged == false) && (!has(c.securityContext.allowPrivilegeEscalation) || c.securityContext.allowPrivilegeEscalation == false)))"
    message: "init containers may not be privileged or allow privilege escalation"
  - expression: "!has(object.spec.ephemeralContainers) || object.spec.ephemeralContainers.all(c, !has(c.securityContext) || ((!has(c.securityContext.privileged) || c.securityContext.privileged == false) && (!has(c.securityContext.allowPrivilegeEscalation) || c.securityContext.allowPrivilegeEscalation == false)))"
    message: "ephemeral containers may not be privileged or allow privilege escalation"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: production-pod-baseline
spec:
  policyName: production-pod-baseline
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        security.example.com/tier: production

This policy activates when controller creates the Pod. To fail earlier at Deployment/Job submit time, add equivalent validations on spec.template.spec of Deployment, StatefulSet, DaemonSet, Job, CronJob. Kyverno simplifies multi-object template validation; with VAP, create separate policies per object type and verify with --dry-run=server. Don't mistake the Pods-only example for pre-deployment validation.

External admission webhooks need HA replicas, PDB, monitoring, short timeouts. failurePolicy: Fail is security-first but makes policy service outage affect matched requests; exclude only necessary bootstrap objects, never make entire kube-system a permanent policy blind spot.

5. Network & Secrets: Close Lateral Paths Beyond API

RBAC won't stop a compromised pod from accessing another pod's port 3306. NetworkPolicy requires CNI that actually implements its semantics. Confirm CNI support, then roll out default-deny per namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-order
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

After enabling, explicitly allow DNS, required API server/cloud metadata access, dependent services, and observability paths; otherwise apps break due to DNS or egress dependency loss. Example DNS allow (verify namespaceSelector labels match your cluster):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: team-order
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Secret least privilege isn't just "no get/list/watch secrets ". Also enable etcd encryption at rest (self-managed), integrate external key management with rotation, avoid exposing high-sensitivity secrets via env vars, and audit secret reads at Metadata level to keep values out of audit logs.

6. Audit & Validation: Make Permissions a Testable Contract

Audit rules match in order; first match decides level. Log only metadata for Secrets; log request body for RBAC changes; final fallback to controlled metadata:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived]
rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
- level: Request
  resources:
  - group: rbac.authorization.k8s.io
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
- level: Metadata
  users: ["system:serviceaccount:ci:deploy-agent"]
- level: Metadata

Self-managed control planes configure via --audit-policy-file; managed clusters enable control-plane audit logs and export to central platform. Production rollout needs retention, access control, alerting, and cost budget.

Test every RBAC change in CI both ways:

# Positive: release identity can deploy
test "$(kubectl auth can-i patch deployments \
  --as=system:serviceaccount:ci:deploy-agent -n team-order)" = yes

# Negative: it cannot read secrets or modify RBAC
test "$(kubectl auth can-i get secrets \
  --as=system:serviceaccount:ci:deploy-agent -n team-order)" = no
test "$(kubectl auth can-i create rolebindings \
  --as=system:serviceaccount:ci:deploy-agent -n team-order)" = no
test "$(kubectl auth can-i create clusterrolebindings \
  --as=system:serviceaccount:ci:deploy-agent)" = no

The operator running these --as commands needs impersonate permission; they are regression tests, not substitutes for end-to-end tests with real CI credentials.

Validate admission policies pre-merge:

kubectl apply --dry-run=server -f deployment.yaml

In staging, exercise two failure modes: valid deployments must pass; manifests with privileged: true, hostPath, or wrong serviceAccountName must fail.

7. Phased Implementation Roadmap

Week 1: Stop the Bleeding

List all cluster-admin bindings; tag each with business owner, purpose, and removal date.

Delete shared admin kubeconfig; stop issuing long-lived ServiceAccount tokens.

Review identities that can read Secrets, write RBAC, bind, escalate, impersonate, pods/exec, and create workloads.

Enable audit and alert on above high-risk actions.

kubectl get clusterrolebindings -o json | jq -r 
  '.items[] | select(.roleRef.name == "cluster-admin") |
  {name: .metadata.name, subjects: .subjects}'

Weeks 2-4: Establish Baseline

Integrate IdP/cloud-native identity; bind permissions to groups, not individuals.

Create viewer, developer, deployer, operator, security-auditor templates.

Migrate CI to dedicated identity + short-lived credentials + target-namespace RoleBinding.

Roll out PSA in warn/audit → remediate → enforce sequence.

Apply NetworkPolicy to critical production namespaces; observe dependencies before tightening.

Continuous Governance

Daily/weekly scan and review: cluster-admin, Secret reads, RBAC writes, ServiceAccount usage, long-lived tokens, temporary grant expiry. High-privilege ops use Break Glass: approval, short-lived grant, full audit, auto-revocation. Goal isn't "no high privilege" but ensuring high privilege cannot persist silently.

Conclusion: Security Maturity Measured by Blast Radius

An RCE can't be guaranteed never to happen; reliable access control ensures:

One CI or workload pod compromised
≠
Attacker gains entire Kubernetes cluster

For every identity, platform team must answer: who it is, when credentials expire, which APIs it can call, which namespaces it can affect, whether it can read Secrets, choose high-privilege ServiceAccount, modify RBAC, whether its created workloads are admission-constrained, whether it can move laterally, and whether high-risk actions are traceable.

When these questions have verifiable answers, Kubernetes access control graduates from "a few YAML snippets" to a production security system.