Verifying Signed Container Images with CRI‑O and Kubernetes v1.28

Kubernetes image and binary signing

Kubernetes started signing container‑image artifacts in version v1.24 and extended signing to binary artifacts (beta) in v1.26. Projects that belong to the kubernetes or kubernetes-sigs GitHub organizations can generate signatures in CI/CD pipelines (e.g., GitHub Actions) or by submitting pull requests to the k/k8s.io repository.

Signature verification approaches

The official documentation describes a manual cosign verify workflow, which is useful only for testing because it requires human interaction. In production, the sigstore policy‑controller provides automated verification via a Custom Resource Definition (CRD), an admission controller, and a webhook that expose a high‑level API for signature checks.

The typical admission‑controller verification flow is illustrated below:

Why a CRI‑based solution?

The admission‑controller model validates an image once, before the kubelet pulls it. Because the node that pulls the image may differ from the node that runs the admission controller, a compromised controller cannot enforce cluster‑wide policies. Performing policy evaluation directly in a CRI‑compatible container runtime eliminates this separation.

CRI‑O will support full image‑signature verification in Kubernetes v1.28.

CRI‑O signature policy

CRI‑O reads a policy.json file that defines rules for container images. An example policy that only allows signed images from quay.io/crio/signed is shown below:

{
  "default": [{ "type": "reject" }],
  "transports": {
    "docker": {
      "quay.io/crio/signed": [
        {
          "type": "sigstoreSigned",
          "signedIdentity": { "type": "matchRepository" },
          "fulcio": {
            "oidcIssuer": "https://github.com/login/oauth",
            "subjectEmail": "[email protected]",
            "caData": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t..."
          },
          "rekorPublicKeyData": "LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0K..."
        }
      ]
    }
  }
}

To apply the policy globally, start CRI‑O with the --signature-policy flag:

sudo crio --log-level debug --signature-policy ./policy.json

Testing the policy

Pull a signed image and observe debug logs confirming successful verification:

sudo crictl -D pull quay.io/crio/signed
DEBU[…] IsRunningImageAllowed for image docker:quay.io/crio/signed:latest
DEBU[…] Requirement 0: allowed
DEBU[…] Overall: allowed

Changing the subjectEmail to an invalid address makes the pull fail:

jq '.transports.docker."quay.io/crio/signed"[0].fulcio.subjectEmail = "[email protected]"' policy.json > new-policy.json
mv new-policy.json policy.json
sudo crictl pull quay.io/crio/signed
FATA[…] Source image rejected: Required email [email protected] not found (got []string{"[email protected]"})

Testing an unsigned image can be done by renaming the key in the policy (e.g., quay.io/crio/signed → quay.io/crio/unsigned) and pulling the image, which results in a SignatureValidationFailed error.

Docker‑reference matching

CRI‑O matches the .critical.identity.docker-reference field in the signature with the image repository. For example, verifying registry.k8s.io/kube-apiserver-amd64:v1.28.0-alpha.3 yields the reference registry.k8s.io/kube-apiserver-amd64:.

New error code in Kubernetes v1.28

Kubernetes v1.28 introduces the SignatureValidationFailed error code for image‑pull failures. A pod that requests an unsigned image will surface this error via kubectl:

apiVersion: v1
kind: Pod
metadata:
  name: pod
spec:
  containers:
  - name: container
    image: quay.io/crio/unsigned

kubectl apply -f pod.yaml
pod/pod created
kubectl get pods
NAME   READY   STATUS                      RESTARTS   AGE
pod    0/1     SignatureValidationFailed   0          4s

Namespace‑specific policies (upcoming)

CRI‑O will soon support --signature-policy-dir, allowing a separate policy file per namespace (e.g., <SIGNATURE_POLICY_DIR>/<NAMESPACE>.json). If a namespace‑specific file is missing, the global policy is used as a fallback.

Edge cases and practical considerations

kubelet only extracts an image when it is not already present on disk. A permissive policy in one namespace can cache the image, preventing stricter policies in other namespaces from taking effect.

CRI‑O must verify signatures both at image extraction and at container creation. The CRI receives an image ID or digest rather than the original reference, so a small change to the CRI is required to pass the original reference for policy evaluation.

Combining the sigstore policy‑controller CRD with CRI‑O’s native verification provides a Kubernetes‑native experience without installing third‑party software in the cluster.