Avoid Night‑Shift Disasters: Lessons from a Kubernetes RBAC Mishap

Kubernetes Production Pitfalls: A Hard‑Earned Lesson from a Permission Misconfiguration

Prologue: The 3 AM Call

Remember that unforgettable early‑morning call: "K8s cluster abnormal, unauthorized access!" An intern’s RBAC mistake almost deleted core services in production, highlighting that Kubernetes security hardening is mandatory, not optional.

1. RBAC: Did You Configure It Correctly?

1.1 The Art of Least‑Privilege

Many teams mistakenly grant cluster-admin to simplify things, which is like giving a temporary worker the keys to every room.

Practical configuration example:

# Development environment - developer role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: developer-role
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "update", "patch"]
---
# Production environment - ops manager role (hierarchical)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ops-manager
rules:
- apiGroups: [""]
  resources: ["nodes", "namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"] # allow only in emergencies

1.2 Proper Use of ServiceAccounts

Using the default ServiceAccount is extremely dangerous. Create a dedicated ServiceAccount for each application:

# Create a dedicated ServiceAccount for the app
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-reader
  namespace: production
---
# Bind minimal necessary permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
subjects:
- kind: ServiceAccount
  name: app-reader
  namespace: production

1.3 Permission Auditing: Know Who Does What

After configuring RBAC, verify permissions with these commands:

# Check if a user can delete pods
kubectl auth can-i delete pods --as=developer -n production
# List all permissions of a user
kubectl auth can-i --list --as=developer -n dev
# Audit RBAC for high‑privilege bindings
#!/bin/bash
echo "=== Checking high‑privilege role bindings ==="
kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name + ": " + (.subjects[]|.name)'

2. Network Policies: Building a Zero‑Trust Network

2.1 Default‑Deny All Traffic

First close the door, then open the windows. Deny all traffic by default and then allow only necessary communication.

# Default deny all inbound traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

2.2 Fine‑Grained Traffic Control

Real‑world scenario: frontend pods can only access the backend API, backend can only access the database, and the database only accepts connections from the backend.

# Frontend → Backend API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
---
# Backend → Database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend
    - namespaceSelector:
        matchLabels:
          name: production
    ports:
    - protocol: TCP
      port: 3306

2.3 Cross‑Namespace Communication Control

Production environments should never be accessed from development environments.

# Allow only specific namespaces to access the logging service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cross-namespace-policy
  namespace: shared-services
spec:
  podSelector:
    matchLabels:
      app: logging-service
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          environment: production
    - namespaceSelector:
        matchLabels:
          environment: staging
    ports:
    - protocol: TCP
      port: 9200

3. Hands‑On Experience: Past Pitfalls

3.1 RBAC Misconfiguration Caused Outage

Scenario: A developer needed to view production logs; excessive permissions led to accidental deletion of a critical ConfigMap.

Use read‑only permissions for log access.

All production changes must go through CI/CD, not direct kubectl.

Regularly audit and revoke temporary permissions.

3.2 NetworkPolicy Misconfiguration Caused Disruption

Scenario: Forgetting to allow DNS (port 53) broke service name resolution.

Correct configuration:

# Allow DNS resolution
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53

3.3 Monitoring and Alerting

After hardening, set up monitoring for abnormal access attempts and runtime security:

# Monitor unauthorized API server access attempts
kubectl logs -n kube-system kube-apiserver-master | grep "Unauthorized"
# Deploy Falco for runtime security monitoring
helm install falco falcosecurity/falco \
  --set falco.grpc.enabled=true \
  --set falco.grpcOutput.enabled=true

4. Security Hardening Checklist

RBAC Checklist

Remove all unnecessary cluster-admin bindings.

Create separate ServiceAccounts for each application.

Enforce the principle of least privilege.

Periodically audit permission assignments.

Disable anonymous access.

Enable audit logs.

NetworkPolicy Checklist

Implement a default‑deny policy.

Restrict cross‑namespace communication.

Protect system components (kube-system).

Allow necessary DNS resolution.

Restrict egress traffic to known services.

Regularly test policy effectiveness.

Additional Security Measures

Enable Pod Security Standards.

Use admission controllers (OPA/Gatekeeper).

Keep Kubernetes versions up‑to‑date.

Scan container images for vulnerabilities.

Encrypt etcd data.

Use network encryption (TLS/mTLS).

5. Automated Security Compliance Checks

Example Bash script that audits high‑privilege accounts, default ServiceAccount usage, missing NetworkPolicies, and privileged containers:

#!/bin/bash
# K8s security audit script

echo "======= K8s Security Audit ======="

echo "[*] Checking cluster-admin bindings..."
kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name'

echo "[*] Checking default ServiceAccount usage..."
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.serviceAccount=="default") | .metadata.namespace + "/" + .metadata.name'

echo "[*] Namespaces without NetworkPolicy..."
for ns in $(kubectl get ns -o name | cut -d/ -f2); do
  policies=$(kubectl get networkpolicy -n $ns 2>/dev/null | wc -l)
  if [ $policies -eq 0 ]; then
    echo "  - $ns: No NetworkPolicy found!"
  fi
done

echo "[*] Checking privileged containers..."
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.spec.containers[].securityContext.privileged==true) | .metadata.namespace + "/" + .metadata.name'

6. Recommended Tools

Kubescape – YAML security scanning.

Polaris – Configuration best‑practice checks.

Kube‑bench – CIS benchmark verification.

Conclusion: Security Is a Marathon

Kubernetes security hardening is an ongoing process. Start with RBAC, then gradually introduce NetworkPolicies, automate checks, and regularly practice incident response. Remember, the cost of a security breach far exceeds the investment in preventive measures.