Kubernetes Service-to-Service Auth: Using ServiceAccount Tokens with TokenReview in Code

Services deployed inside Kubernetes need a way to verify the identity of callers. When the orders service calls the inventory service, the inventory service must confirm the request originates from the orders service. One approach is to assign each service a dedicated ServiceAccount and use Kubernetes-issued tokens as proof of identity. After configuring ServiceAccounts, the application must still read the token, send it with requests, verify the peer's token, and then decide whether the caller is authorized to access the endpoint.

Distinguishing Identity, Credential, and Permission

A ServiceAccount represents the workload's identity. The example creates two ServiceAccounts, orders and inventory, in the demo namespace. These are distinct from Kubernetes Services of the same name: Services handle traffic routing, while ServiceAccounts handle identity.

The token is the credential that proves identity. The article uses short-lived ServiceAccount tokens projected into the pod's filesystem. These are signed JWTs containing issuer, subject, expiration, and audience claims. The JWT payload can be decoded; the signature prevents tampering, but the content is not encrypted.

After successful verification, the inventory service receives a caller identity like system:serviceaccount:demo:orders. This answers "which ServiceAccount's credential was presented" but does not automatically answer "is this identity allowed to decrement stock". The latter requires the inventory service's own authorization rules. The example stipulates that only the orders ServiceAccount may call GET /stock; even another service with a valid token would be denied. Additionally, multiple pods sharing the orders ServiceAccount are treated as a single identity, and this does not represent the end user. If per-user, per-tenant, or per-order authorization is needed, additional business-level identity and permission checks must be added.

Why a Single Request Involves Two Tokens

The business request from orders to inventory carries the orders service's token (audience: inventory). Upon receipt, the inventory service calls the Kubernetes API Server's TokenReview endpoint to validate that token, using its own ServiceAccount token (audience: Kubernetes API Server) for authentication. The two requests therefore use different identities:

Orders → Inventory : Authorization header bears the orders token, audience urn:demo:inventory.

Inventory → API Server : Authorization header bears the inventory service's own API token; the request body's spec.token field contains the orders token to be validated.

This distinction is the most common source of confusion. The inventory service must not reuse the orders token as its own credential when calling the Kubernetes API.

A successful TokenReview only completes identity verification. The inventory service must then read the returned identity and evaluate its local authorization rules before executing the inventory query.

Step 1: Mount a Dedicated Token for the Caller

First, create the identities:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders
  namespace: demo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: inventory
  namespace: demo

The orders pod spec fragment below uses a projected volume to mount a token intended for the inventory service. When using a Deployment, place this under spec.template.spec and merge with existing container configuration. Replace the image address as needed.

serviceAccountName: orders
automountServiceAccountToken: false
containers:
  - name: orders
    image: your-registry/orders:demo
    volumeMounts:
      - name: inventory-token
        mountPath: /var/run/secrets/inventory
        readOnly: true
volumes:
  - name: inventory-token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: urn:demo:inventory
            expirationSeconds: 3600

automountServiceAccountToken: false

disables the default API token mount but does not prevent the explicitly declared projected volume. The orders service only needs to call the inventory API, so it does not require Kubernetes resource permissions. The projected token is fetched by the kubelet; the application does not need to call TokenRequest itself.

The audience field indicates the intended recipient of the credential. Here urn:demo:inventory identifies the inventory service; it need not be a resolvable URL, but the issuer and verifier configurations must match exactly.

Audience restriction cannot replace caller authorization. A token with aud set to the inventory service does not prove it came from the orders service; the identity must still be checked. expirationSeconds: 3600 requests a one-hour validity; the actual lifetime is bounded by cluster configuration. The kubelet rotates projected tokens, so the application must re-read the file. Do not bake tokens into images, ConfigMaps, or long-lived environment variables, and do not use subPath to mount a file that requires updates.

Step 2: Read the Token File and Add a Bearer Header

The orders service's request code reads the token file on each call and sends it as a Bearer token. A custom NoRedirect handler prevents credentials from being sent to unexpected redirects.

import ssl
from pathlib import Path
from urllib.request import (
    HTTPRedirectHandler, HTTPSHandler, Request, build_opener,
)

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

def get_stock():
    token = Path(
        "/var/run/secrets/inventory/token"
    ).read_text().strip()
    context = ssl.create_default_context(cafile="/trust/ca.crt")
    opener = build_opener(HTTPSHandler(context=context), NoRedirect())
    request = Request(
        "https://inventory.demo.svc:8443/stock",
        headers={"Authorization": f"Bearer {token}"},
    )
    with opener.open(request, timeout=5) as response:
        return response.read().decode()

Reading the token file on every call is the simplest rotation handling for teaching purposes. High-throughput scenarios can reuse connections and TLS contexts with a short refresh cycle, but must never read the token only once at process start.

The example assumes the inventory service runs HTTPS with a certificate covering inventory.demo.svc, and the orders service mounts the issuing CA at /trust/ca.crt. This CA may differ from the Kubernetes API Server's CA.

ServiceAccount tokens are Bearer credentials; anyone possessing them can use them. Using plain HTTP just because traffic stays inside the cluster would expose credentials; signatures do not prevent replay after interception.

Step 3: Allow the Inventory Service to Call TokenReview

The inventory service needs permission to create TokenReview resources. The following RBAC grants only that permission:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: demo-inventory-token-reviewer
rules:
  - apiGroups: ["authentication.k8s.io"]
    resources: ["tokenreviews"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: demo-inventory-token-reviewer
subjects:
  - kind: ServiceAccount
    name: inventory
    namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: demo-inventory-token-reviewer

TokenReview is a cluster-scoped API, hence the ClusterRole and ClusterRoleBinding. create means initiating a verification, not persisting a business object in etcd.

The inventory pod uses the default ServiceAccount token for API Server access:

serviceAccountName: inventory
automountServiceAccountToken: true

In a standard cluster, the default volume provides the token and CA file at:

/var/run/secrets/kubernetes.io/serviceaccount/token
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

This default token is for the inventory service to call the Kubernetes API; it is distinct from the urn:demo:inventory token mounted in the orders pod.

Do not grant cluster-admin to the inventory service for authentication, and do not give the orders service tokenreviews permission. In this example, only the inventory service performs verification.

Step 4: Verify Identity in Service Code

The TokenReview request sent to the API Server has the following structure (placeholders in angle brackets):

POST /apis/authentication.k8s.io/v1/tokenreviews
Authorization: Bearer <inventory's own API token>
Content-Type: application/json

{
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenReview",
  "spec": {
    "token": "<orders token>",
    "audiences": ["urn:demo:inventory"]
  }
}

The Python module auth.py implements the verification. It uses HTTPS with API Server certificate validation and re-reads the inventory service's own token on each call to respect rotation.

import json
import ssl
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import (
    HTTPRedirectHandler, HTTPSHandler, Request, build_opener,
)

AUDIENCE = "urn:demo:inventory"
SA_DIR = Path("/var/run/secrets/kubernetes.io/serviceaccount")
REVIEW_URL = (
    "https://kubernetes.default.svc"
    "/apis/authentication.k8s.io/v1/tokenreviews"
)

class AuthFailure(Exception):
    def __init__(self, status):
        self.status = status

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

def review_token(token):
    try:
        reviewer_token = (SA_DIR / "token").read_text().strip()
        context = ssl.create_default_context(
            cafile=str(SA_DIR / "ca.crt")
        )
        opener = build_opener(
            HTTPSHandler(context=context), NoRedirect()
        )
        body = {
            "apiVersion": "authentication.k8s.io/v1",
            "kind": "TokenReview",
            "spec": {"token": token, "audiences": [AUDIENCE]},
        }
        request = Request(
            REVIEW_URL,
            data=json.dumps(body).encode(),
            headers={
                "Authorization": f"Bearer {reviewer_token}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        with opener.open(request, timeout=3) as response:
            result = json.load(response)
    except (OSError, HTTPError, URLError, ValueError):
        raise AuthFailure(503) from None
    if (
        not isinstance(result, dict)
        or not isinstance(result.get("status"), dict)
    ):
        raise AuthFailure(503)
    return result["status"]

The 503 status indicates the authentication dependency is unavailable; business logic must not proceed. Real systems should log sanitized diagnostics (timeouts, upstream status codes) but never log Authorization headers, full tokens, or TokenReview request bodies.

If the API Server returns HTTP 403, first verify the inventory service has create tokenreviews permission. This is separate from a business 403 due to the orders service lacking permission for the inventory endpoint.

A successful TokenReview request does not mean the presented token is valid. You must check status.authenticated is true and that status.audiences includes the expected audience.

Step 5: Perform Authorization After Authentication

Add the following authorization logic to auth.py:

ALLOWED = {
    ("system:serviceaccount:demo:orders", "GET", "/stock")
}

def authorize(header, method, path, review=review_token):
    parts = header.split()
    if len(parts) != 2 or parts[0].lower() != "bearer":
        raise AuthFailure(401)
    token = parts[1]
    if len(token) > 16384:
        raise AuthFailure(401)

    status = review(token)
    if status.get("error"):
        raise AuthFailure(503)
    if status.get("authenticated") is not True:
        raise AuthFailure(401)
    audiences = status.get("audiences")
    if not isinstance(audiences, list) or AUDIENCE not in audiences:
        raise AuthFailure(401)

    user = status.get("user")
    username = user.get("username") if isinstance(user, dict) else None
    if not isinstance(username, str):
        raise AuthFailure(401)
    if (username, method, path) not in ALLOWED:
        raise AuthFailure(403)
    return username

The 16384 limit is an example input cap, not a Kubernetes token length specification. Production should also enforce request header size limits at the HTTP entry point.

Treating any non-empty status.error as 503 is a conservative example strategy. This field may also contain validation errors like expired tokens; it cannot reliably distinguish API Server failure from token invalidity. Hence an expired token may also yield 503; the security requirement is to refuse business execution, not to guarantee all invalid tokens return 401. Do not rely on error strings for stable classification or unlimited retries.

The flow diagram shows: authentication failure → 401; valid identity but unauthorized → 403; verification dependency failure → 503; none of these proceed to business logic.

In a web framework, place authorize() in middleware, a dependency function, or a pre-route check. The processing order:

Accept exactly one Authorization header; reject duplicates.

Call authorize(header, method, path).

Catch AuthFailure, return its status code; for 401 include WWW-Authenticate: Bearer.

Only after successful return execute data queries or mutations.

The accompanying server.py provides a standard-library HTTPS server example, and client.py is the caller. They reside in the same directory as auth.py and run on Python 3.11+. The server requires /tls/tls.crt and /tls/tls.key, listens on 8443, and the Kubernetes Service should forward traffic to that port. Certificate issuance, Secret/CA mounting, and image building must be completed per environment. The standard library HTTP server is for teaching only; production should use a framework with concurrency limits, timeouts, and observability.

The example matches /stock exactly; /stock?sku=1 would be rejected. Real frameworks should authorize using the resolved route or explicit resource actions, and validate parameters; do not widen allowed paths by stripping segments arbitrarily.

Authorization here checks only the ServiceAccount fully qualified name. If the account is deleted and recreated with the same name, the rule would still allow the new account. To bind to a specific account instance, also verify the user.uid returned by TokenReview.

Clarifying "Mutual Authentication"

In the described flow, the inventory service verifies the orders identity via token, while the orders service verifies the inventory server via HTTPS certificate. The orders service does not read or verify the inventory service's ServiceAccount token.

If "mutual authentication" means both services call each other, configure symmetrically:

Orders → Inventory: send orders token with audience urn:demo:inventory.

Inventory → Orders: send inventory token with audience urn:demo:orders.

Both sides implement TokenReview verification and their own business authorization, and both need corresponding reviewer permissions.

A single pod can mount multiple tokens for different targets; the API Server credential remains separate. Avoid stuffing multiple purposes into one generic token.

If the requirement is "both parties prove workload identity on the same connection", evaluate mTLS. Attaching a token in the response is not equivalent, nor is a plain Bearer request equivalent to mTLS.

Can We Skip TokenReview and Verify the JWT Locally?

Yes. An alternative path fetches the JWKS (JSON Web Key Set) from the trusted cluster's issuer discovery endpoint and uses a mature JWT library for local verification.

However, merely decoding the JWT and reading the sub claim is insufficient. Local verification must check the signature, pin the trusted issuer, allow only approved algorithms, validate audience and time constraints, and handle key rotation. Never fetch an issuer URL from an unverified token.

The main differences lie in dependencies and revocation semantics: TokenReview adds API Server load but can check the current state of the bound object; offline verification reduces control-plane dependency at verification time but cannot know if the Pod or ServiceAccount has been deleted, potentially accepting tokens until expiry.

For initial understanding and implementation, the TokenReview path is recommended. When optimizing throughput, measure call volume, latency, and control-plane capacity before introducing caching or offline verification. Caches must include audience as a dimension; cache keys can use token hashes; TTLs should have a short ceiling and be bounded by remaining token lifetime, accepting the resulting revocation delay. Authentication dependency failures must never default to allow.

Pre-Production Failure Path Verification

The article has not been run end-to-end in a real cluster. The following acceptance criteria must be validated in an isolated test environment:

Correct identity and audience accessing GET /stock returns 200.

Missing token, forged token, expired token, or wrong audience are rejected.

Another ServiceAccount with a valid token for the correct audience still receives business 403.

Removing the inventory service's reviewer permission or breaking its API Server connectivity prevents business execution; the example returns 503.

Continuous operation across a token rotation still succeeds.

After deleting the bound Pod, verify the old token is rejected after grace period and cache windows expire.

Using an incorrect server certificate or untrusted CA causes client TLS verification to fail.

Also protect "who can assume these identities". If a user can create a pod using the orders ServiceAccount in the target namespace, or request a token for it, they may obtain the orders service identity. Application-layer allowlists must be built on trusted workload creation permissions.

This scheme delegates credential issuance and rotation to Kubernetes, but service code retains clear responsibilities: the caller reads the latest credential, the callee verifies audience and identity, then enforces interface-level authorization. Separating these actions makes ServiceAccount tokens a transparent, auditable service-to-service authentication mechanism.