Secure Secrets: Install & Integrate HashiCorp Vault with Kubernetes

Installation

Installing on a Linux host

Installation on a Linux host is straightforward and requires three steps:

# Install package manager tools
$ sudo yum install -y yum-utils
# Add repository
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# Install vault
$ sudo yum -y install vault

Installing in Kubernetes

Vault provides a Helm chart that can be installed with Helm.

Version requirements: Helm 3.0+ Kubernetes 1.9+

# Add Helm repo
$ helm repo add hashicorp https://helm.releases.hashicorp.com
# Update local repo
$ helm repo update
# Install vault
$ helm install vault hashicorp/vault

Starting the Vault server

After installing Vault on a host, you can start a development server (not for production) with the following command:

$ vault server -dev -dev-listen-address=0.0.0.0:8200 &
# ...
WARNING! dev mode is enabled! In this mode, Vault runs entirely in‑memory and starts unsealed with a single unseal key. The root token is already authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://0.0.0.0:8200'

Unseal Key: killR+cPfTR7P7HoYRt5SsMySMDv2w9WD7ljcxpXB+Q=
Root Token: s.pd4FBsC1pamE21nLv3fszdI1

Development mode should NOT be used in production installations.

Access the UI at http://<em>IP</em>:8200/ui and log in with the generated token.

Configuring Kubernetes to communicate with Vault

To allow Kubernetes to read secrets from Vault, ensure the two systems can communicate and configure the Kubernetes auth method.

# Set Vault address
export VAULT_ADDR=http://192.168.0.153:8200

# Enable Kubernetes auth method
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

# Configure Kubernetes auth
$ vault write auth/kubernetes/config \
    kubernetes_host=https://192.168.0.153:6443 \
    kubernetes_ca_cert=@/etc/kubernetes/pki/ca.crt
Success! Data written to: auth/kubernetes/config

# Create a policy (demo)
$ cat <<EOF | vault policy write vault-demo-policy -
path "sys/mounts" { capabilities = ["read"] }
path "secret/data/demo/*" { capabilities = ["read"] }
path "secret/metadata/demo/*" { capabilities = ["list"] }
EOF
Success! Uploaded policy: vault-demo-policy

# Create a role bound to a service account
$ vault write auth/kubernetes/role/vault-demo-role \
    bound_service_account_names=vault-serviceaccount \
    bound_service_account_namespaces=default \
    policies=vault-demo-policy \
    ttl=1h
Success! Data written to: auth/kubernetes/role/vault-demo-role

Storing a secret in Vault

$ vault kv put secret/demo/database username="coolops" password=123456
Key              Value
---              -----
created_time     2021-01-25T08:22:35.134166877Z
deletion_time    n/a
destroyed        false
version          1

# Retrieve the secret
$ vault kv get secret/demo/database
====== Metadata ======
Key              Value
---              -----
created_time     2021-01-25T08:22:35.134166877Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key        Value
---        -----
password   123456
username   coolops

Creating RBAC permissions in Kubernetes

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-serviceaccount
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: vault-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-serviceaccount
  namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secretadmin-role
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secretadmin-rolebinding
subjects:
- kind: ServiceAccount
  name: vault-serviceaccount
roleRef:
  kind: Role
  name: vault-secretadmin-role
  apiGroup: rbac.authorization.k8s.io

Apply the RBAC configuration:

$ kubectl apply -f rbac.yaml
serviceaccount/vault-serviceaccount created
clusterrolebinding.rbac.authorization.k8s.io/vault-clusterrolebinding created
role.rbac.authorization.k8s.io/vault-secretadmin-role created
rolebinding.rbac.authorization.k8s.io/vault-secretadmin-rolebinding created

Using Vault secrets in Kubernetes

There are two ways to retrieve secrets from Vault inside a pod:

Use the Vault Agent in an initContainer to fetch the secret.

Use the Vault SDK directly in application code.

InitContainer approach

A ConfigMap holds the Vault Agent configuration:

<code>apiVersion: v1
data:
  vault-agent-config.hcl: |
    exit_after_auth = true
    pid_file = "/home/vault/pidfile"
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "vault-demo-role"
        }
      }
      sink "file" {
        config = { path = "/home/vault/.vault-token" }
      }
    }
    template {
      destination = "/etc/secrets/index.html"
      contents = <<EOT
      <html>
      <body>
      <p>Some secrets:</p>
      {{- with secret "secret/demo/database" }}
      <ul>
        <li><pre>username: {{ .Data.data.username }}

password: {{ .Data.data.password }}

{{ end }}

EOT } kind: ConfigMap metadata: name: example-vault-agent-config namespace: default

Pod definition using the initContainer:

apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-serviceaccount
  volumes:
  - name: config
    configMap:
      name: example-vault-agent-config
      items:
      - key: vault-agent-config.hcl
        path: vault-agent-config.hcl
  - name: shared-data
    emptyDir: {}
  initContainers:
  - name: vault-agent
    image: registry.cn-hangzhou.aliyuncs.com/rookieops/vault:1.6.1
    args: ["agent", "-config=/etc/vault/vault-agent-config.hcl", "-log-level=debug"]
    env:
    - name: VAULT_ADDR
      value: http://192.168.0.153:8200
    volumeMounts:
    - name: config
      mountPath: /etc/vault
    - name: shared-data
      mountPath: /etc/secrets
  containers:
  - name: nginx-container
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html

After the pod runs, the secret can be accessed via the web server:

<code>$ curl 172.16.235.231
<html>
<body>
<p>Some secrets:</p>
<ul>
<li><pre>username: coolops

password: 123456

SDK approach

package main

import (
    "fmt"
    "io/ioutil"
    vaultApi "github.com/hashicorp/vault/api"
)

var (
    vaultHost           string
    vaultCAPath         string
    vaultServiceAccount string
    vaultJWTPath        string
)

func main() {
    // Kubernetes token path
    vaultJWTPath = "/var/run/secrets/kubernetes.io/serviceaccount/token"
    // ServiceAccount name
    vaultServiceAccount = "vault-serviceaccount"

    tlsConfig := &vaultApi.TLSConfig{CACert: vaultCAPath, Insecure: false}
    config := vaultApi.DefaultConfig()
    config.Address = fmt.Sprintf("https://%s", vaultHost)
    config.ConfigureTLS(tlsConfig)

    client, _ := vaultApi.NewClient(config)
    buf, _ := ioutil.ReadFile(vaultJWTPath)
    jwt := string(buf)

    options := map[string]interface{}{"jwt": jwt, "role": vaultServiceAccount}
    loginSecret, _ := client.Logical().Write("auth/kubernetes/login", options)
    client.SetToken(loginSecret.Auth.ClientToken)

    secret, _ := client.Logical().Read("database/creds/tx")
    fmt.Println(secret)
}

Conclusion

Vault is a powerful tool for securely managing sensitive information. Although the configuration steps are relatively complex and require careful maintenance, integrating Vault with Kubernetes remains an effective solution for secret management.

References:

https://github.com/hashicorp/vault

https://github.com/hashicorp/vault-helm

https://www.vaultproject.io/docs/agent

https://www.vaultproject.io/docs/agent/template

https://learn.hashicorp.com/tutorials/vault/agent-kubernetes

https://medium.com/getamis/vault-kubernetes-integration-63ce46d47550