How to Secure Kubernetes Secrets: Static Encryption and KMS Plugin Guide

Why encrypt?

In Kubernetes, Secrets store sensitive data such as passwords and certificates, but by default they are only base64‑encoded, which can be easily decoded to reveal the original values.

For example, creating a secret with:

<code>echo -n "coolops" | kubectl create secret generic mysecret --dry-run --from-file=secret=/dev/stdin -o yaml > secret.yaml
cat secret.yaml
apiVersion: v1
data:
  secret: Y29vbG9wcw==
kind: Secret
metadata:
  name: mysecret
</code>

Anyone with access to the secret value can decode it:

<code>echo "Y29vbG9wcw==" | base64 -d
coolops
</code>

This is insecure. In a Kubernetes cluster, etcd stores all resources, including Secrets, so compromising etcd gives full access to the cluster; therefore encrypting Secrets in production is essential.

How to encrypt?

Static encryption

Since Kubernetes 1.13, static encryption can be enabled via the kube‑apiserver, which stores encrypted data in etcd. Attackers who obtain etcd cannot read the original Secrets.

Current cluster installed with kubeadm, version 1.18.9

Steps:

Create an encryption configuration file (e.g.,

/etc/kubernetes/pki/static-secret-encryption.yaml

).

<code>apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: mysecret
              secret: DJqYaMMpY2DNlHz+HYrFYOUSh5SXKWiVOwLf6nQX9ss=
      - identity: {}
</code>

Generate the encryption key with:

<code>head -c 32 /dev/urandom | base64</code>

Modify the kube‑apiserver manifest (

/etc/kubernetes/manifests/kube-apiserver.yaml

) to add the flag

--encryption-provider-config=/etc/kubernetes/pki/static-secret-encryption.yaml

.

Note: the flag is --encryption-provider-config from version 1.14 onward.

Restart the kube‑apiserver.

Verify encryption by creating a secret and inspecting its stored value in etcd. The data should start with

k8s:enc:aescbc:v1:

, indicating it is encrypted with the aescbc provider and the key named

mysecret

.

KMS plugin

The KMS provider uses a envelope‑encryption model: data is encrypted with a data‑encryption key (DEK), which is itself encrypted by a key‑encryption key (KEK) stored in an external KMS. The plugin communicates with the KMS via gRPC.

Example using Alibaba Cloud KMS:

Enable KMS service.

Create a KMS key.

Deploy the ack‑kms‑plugin deployment (manifest shown below).

<code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: ack-kms-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: ack-kms-plugin
  template:
    metadata:
      labels:
        name: ack-kms-plugin
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - preference: {}
              weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node
                    operator: In
                    values:
                      - master
      restartPolicy: Always
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: kmssocket
          hostPath:
            path: /var/run/kmsplugin
            type: DirectoryOrCreate
      containers:
        - name: ack-kms-plugin
          image: registry.{{ .Region }}.aliyuncs.com/acs/ack-kms-plugin:v1.0.2
          imagePullPolicy: Always
          command:
            - ack-kms-plugin
            - --gloglevel=5
            - --key-id={{ .KeyId }}
            - --path-to-unix-socket=/var/run/kmsplugin/grpc.sock
          livenessProbe:
            exec:
              command:
                - ack-kms-plugin
                - health
                - --path-to-unix-socket=/var/run/kmsplugin/grpc.sock
            initialDelaySeconds: 30
            failureThreshold: 3
            timeoutSeconds: 5
            periodSeconds: 300
          env:
            - name: ACCESS_KEY_ID
              value: {{ .AK }}
            - name: ACCESS_KEY_SECRET
              value: {{ .AK_Secret }}
            - name: CREDENTIAL_INTERVAL
              value: {{ .Credential_Interval }}
          volumeMounts:
            - name: kmssocket
              mountPath: /var/run/kmsplugin
              readOnly: false
</code>

Replace placeholders such as

{{ .Region }}

,

{{ .KeyId }}

,

{{ .AK }}

,

{{ .AK_Secret }}

, and

{{ .Credential_Interval }}

with your actual values.

Create a KMS‑based encryption configuration (

/etc/kubernetes/kmsplugin/encryptionconfig.yaml

) that references the plugin socket:

<code>apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: grpc-kms-provider
          endpoint: unix:///var/run/kmsplugin/grpc.sock
          cachesize: 1000
          timeout: 3s
      - identity: {}
</code>

Update the kube‑apiserver manifest to use

--encryption-provider-config=/etc/kubernetes/kmsplugin/encryptionconfig.yaml

and mount the plugin socket and config files:

<code>...spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/kmsplugin/encryptionconfig.yaml
...</code>

<code>...volumeMounts:
  - name: kms-sock
    mountPath: /var/run/kmsplugin
  - name: kms-config
    mountPath: /etc/kubernetes/kmsplugin
...volumes:
  - name: kms-sock
    hostPath:
      path: /var/run/kmsplugin
  - name: kms-config
    hostPath:
      path: /etc/kubernetes/kmsplugin
</code>

Restart the kube‑apiserver. New Secrets will be stored with the envelope‑encryption scheme; the stored value will be prefixed with

k8s:enc:kms:v1:grpc-kms-provider

, confirming that the KMS provider encrypted the data.

Third‑party plugins

Other solutions such as sealed‑secrets or HashiCorp Vault can also provide secret encryption for Kubernetes.

References:

Kubernetes data encryption guide

KMS provider documentation

ACK KMS plugin

HashiCorp Vault

Sealed Secrets