Kubernetes Authentication Time Bomb: The Evolution and Production Practices of ServiceAccount Tokens

Why the incident started before token expiration

A payment platform experienced a sudden 401 Unauthorized error when the order‑service could no longer reach the API server. Initial guesses pointed to RBAC changes, certificate issues, or cached credentials, but the root cause was a legacy Secret -based ServiceAccount token that had been copied into scripts and repositories, and teams disagreed on token rotation responsibilities.

This situation exposed three hidden problems:

Static credentials that never expire remain in the cluster.

Many workloads receive API credentials they don’t need.

Teams assume that "Pods have identity" means the identity management is already secure.

Clarifying ServiceAccount, Token, and default mounting

1. ServiceAccount is the identity subject

It answers "which identity does this Pod run as?" A typical subject looks like system:serviceaccount:payment:order-service, where payment is the namespace and order-service is the ServiceAccount name. RBAC matches this subject and its groups.

2. Token is the credential

Tokens are JWTs. Two implementation paths exist:

Secret‑based token (pre‑v1.24): mounted as a Secret volume, never expires, not auto‑rotated, not recommended.

TokenRequest / projected token (current): mounted as a projected volume, supports expiration, audience, and automatic rotation, and is recommended.

3. Default mounting is an Admission controller behavior

If automountServiceAccountToken is not explicitly disabled, the Admission controller injects a projected token volume. Even without an explicit volume definition, a Pod may already receive API credentials.

Why the old secret‑based approach is a "time bomb"

Legacy tokens lack a natural expiration, are not rotated by kubelet, can be easily copied, and are often treated as permanent cluster credentials. Although Kubernetes v1.24 disables automatic generation of such tokens, existing secrets persist, manual long‑lived tokens can still be created, and external integrations may still rely on them.

Automatic cleanup mechanisms only address unused legacy tokens; manually created tokens and leaked credentials remain vulnerable.

What the new projected token changes

The recommended solution combines the TokenRequest API with a projected ServiceAccount token volume. Key characteristics:

Has an expiration time (default 1 hour, minimum 10 minutes).

Supports a configurable audience.

Kubelet automatically rotates the token before 80% of its lifetime or after 24 hours.

The token is bound to the specific Pod object, so when the Pod is deleted the token is no longer trusted.

This binding adds two operational benefits:

The token is no longer just "an account's credential" but "a credential for a specific object within a time window".

The API server validates the binding during authentication.

Full authentication flow for a request

Parse the token and confirm it is a ServiceAccount credential.

Validate the signature.

Check time constraints ( nbf, iat, exp).

Verify the audience is accepted.

Ensure the bound object (Pod, ServiceAccount) still exists.

Map sub to a user identity (e.g., system:serviceaccount:payment:order-service).

Proceed to RBAC authorization.

A 401 usually indicates a failure before RBAC (e.g., expired token, audience mismatch), while a 403 means the identity is valid but lacks permission.

Production hardening steps

First, classify workloads:

Workloads that never need the API server – set automountServiceAccountToken: false.

Workloads that only need the API – use the default projected token.

Workloads that need both the API and external systems – project separate tokens with distinct audiences.

Example YAML for a minimal ServiceAccount and RBAC:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-service
  namespace: payment
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: order-service-role
  namespace: payment
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["order-service-config"]
  verbs: ["get","watch"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get","create","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: order-service-rb
  namespace: payment
subjects:
- kind: ServiceAccount
  name: order-service
  namespace: payment
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: order-service-role

Key points:

Disable automountServiceAccountToken on the ServiceAccount to avoid unwanted default tokens.

Grant only the exact verbs needed; avoid high‑risk permissions like list secrets.

Explicitly project required tokens in the Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: order-service
  namespace: payment
spec:
  serviceAccountName: order-service
  automountServiceAccountToken: false
  containers:
  - name: app
    image: example.com/order-service:1.0.0
    volumeMounts:
    - name: identity-tokens
      mountPath: /var/run/secrets/tokens
      readOnly: true
  volumes:
  - name: identity-tokens
    projected:
      sources:
      - serviceAccountToken:
          path: kube-api-token
          audience: https://kubernetes.default.svc
          expirationSeconds: 3600
      - serviceAccountToken:
          path: vault-token
          audience: vault
          expirationSeconds: 3600

This separation ensures a leaked vault-token cannot be used to call the API server directly, and each audience can have its own TTL.

Common application mistakes

Developers often treat the token as a constant:

Read the token once at startup and store it in a global variable.

Pass the token via environment variables.

Mount the token file with subPath, which prevents projected volume updates from being seen.

Write a custom HTTP client that reuses a long‑lived token.

Correct practices:

Use the official client SDK, which reads the token file and detects rotation.

If a custom implementation is needed, re‑read the token file on each request or at short intervals.

Avoid subPath for token files because projected volume updates are not reflected.

Go example using InClusterConfig() (which reads the token file automatically):

package main

import (
    "context"
    "log"
    "time"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
)

func main() {
    cfg, err := rest.InClusterConfig()
    if err != nil { log.Fatalf("build in-cluster config failed: %v", err) }
    clientset, err := kubernetes.NewForConfig(cfg)
    if err != nil { log.Fatalf("create client failed: %v", err) }
    ticker := time.NewTicker(30 * time.Second)
    defer ticker.Stop()
    for range ticker.C {
        _, err := clientset.CoreV1().ConfigMaps("payment").Get(context.Background(), "order-service-config", metav1.GetOptions{})
        if err != nil { log.Printf("k8s api call failed: %v", err); continue }
        log.Println("configmap read succeeded")
    }
}

Java example shows reading the token file before each external call.

Migration pitfalls not covered by official docs

Disabling default token injection can break controllers, operators, leader‑election components, or admission webhooks that expect an in‑cluster config. 401 errors often stem from audience mismatch, expired token, or deleted ServiceAccount; 403 errors are usually RBAC permission issues. Treat them separately in monitoring.

External systems that only verify JWT signatures may still accept a token after the Pod is gone; use TokenReview when immediate revocation is required.

Using a single audience for both API server and external services flattens security boundaries; leaked tokens can be used against the API server.

Pre‑deployment checklist

Cluster side

Check for any manually created kubernetes.io/service-account-token Secrets.

Identify workloads still relying on long‑lived tokens.

Verify service-account-issuer, signing keys, and audience settings are consistent.

Ensure monitoring for 401, 403, and TokenReview latency/failure rates.

Workload side

Identify Pods that do not need API access.

Identify Pods that need both API and external trust.

Search for code that reads the token once at startup.

Search for subPath token mounts.

Permission side

Avoid using the default ServiceAccount for production workloads.

Remove overly broad ClusterRoleBinding s.

Eliminate unnecessary list/watch secrets permissions.

Integration side

Confirm external services validate the token audience.

Use TokenReview for scenarios requiring immediate revocation.

Do not equate successful JWT verification with current validity.

Final takeaways

The evolution of ServiceAccount tokens is not just a version bump; it transforms a permanent shared secret into a short‑lived credential tied to a Pod’s lifecycle. Teams should:

Reclaim unnecessary default credentials.

Explicitly project tokens only for workloads that truly need them.

Separate boundaries with distinct audiences and RBAC rules.

Consume tokens as rotatable credentials, not as static constants.

Only after these steps does ServiceAccount management move from "it works" to "it’s governable".