Skip to content
Vault Production Hardening: Dynamic Leases, Raft Tuning & Zero-Disk Secrets
Operations 62 min read

Vault Production Hardening: Dynamic Leases, Raft Tuning & Zero-Disk Secrets

A comprehensive guide to securing HashiCorp Vault in production: dynamic database credentials with automatic lease revocation, Raft cluster tuning formulas, memory-only secret injection via tmpfs, OS-level hardening (mlock, swap disable), audit log resilience, and three real-world incident postmortems with quantitative fixes.

MaGe Linux Operations
MaGe Linux Operations
MaGe Linux Operations

1. Why Static Secrets Fail in Production

Traditional static secrets in .env, Ansible variables, or Kubernetes Secret objects suffer three structural defects:

Credential lifecycle decoupled from permissions : Static passwords lack auto-expiry and on-demand revocation, making blast-radius containment impossible after personnel changes or image leaks.

Missing audit accountability : Shared database accounts across microservices prevent tracing connections to specific Pods or hosts.

Storage and memory exposure : Kubernetes stores Secrets in etcd as plaintext (unless KMS encryption is enabled), and container restarts or disk snapshots can persist credentials on disk.

2. Vault Core Architecture: Encryption Barrier & Dynamic Leases

2.1 Sealed/Unsealed State and Auto-Unseal

Vault starts Sealed ; all data in the storage backend (Integrated Storage Raft) is encrypted with AES-256-GCM . No plaintext root key exists in memory. Any API request returns 503 Service Unavailable. Unseal restores the Encryption Key via cloud KMS/HSM ( Auto-Unseal ), decrypting the stored seal-wrapped keys and re-establishing the in-memory Root Key.

2.2 Dynamic Credential Issuance Loop

Identity verification : Pod presents Kubernetes ServiceAccount JWT ( aud and sub bound to namespace/service-account) to auth/kubernetes/login.

Dynamic issuance : After ACL policy check, Vault's Database Secrets Engine uses a management account to execute CREATE USER and GRANT on PostgreSQL/MySQL, generating a temporary account with a random strong password.

Lease tracking & auto-revocation : The account is bound to a Lease ID (e.g., ttl=1h, max_ttl=24h). Client must renew via sys/leases/renew before TTL/2; on Pod destruction or lease expiry, Vault's expiration manager issues DROP USER.

[App Pod Start]
  | (1) POST /v1/auth/kubernetes/login (SA JWT)
  v
[Vault Server] ---> Verify JWT & Role binding
  | (2) Return Client Token (ttl=1h)
  v
[Vault Agent / App]
  | (3) GET /v1/database/creds/orders-rw-role
  v
[Vault DB Engine] ---> Execute CREATE ROLE + GRANT
  | (4) Return dynamic account + lease_id (ttl=1h, max_ttl=24h)
  v
+-------------------------------------------------------+
| Background renewal goroutine (triggers every 1/2 TTL) |
| PUT /v1/sys/leases/renew -> Extend lease to max_ttl   |
+-------------------------------------------------------+
  | (5) Pod destroyed / renewal timeout / max_ttl reached
  v
[Vault Expiration Manager] ---> Issue DROP ROLE for full cleanup

Engineering caveat : If the business container doesn't gracefully close connections on SIGTERM or reload /vault/secrets/db.env (common in Java/Go services that read env vars once at startup), long-lived connection pools will throw password authentication failed when max_ttl (24h) expires and Vault revokes the old account. Dynamic DB credentials require hot-reload-capable connection pools or short Pod rollout cycles.

2.3 Production Topology: Vault Raft Cluster + K8s Injection

+---------------------------------------------------------------+
| Kubernetes Cluster (10.20.0.0/16)                             |
|  +------------------------+   +-------------------------+     |
|  | App Pod (ns: prod-pay) |   | Vault Agent Injector    |     |
|  | [Init/Sidecar: Agent]  |<-->| MutatingWebhook :8080   |     |
|  | tmpfs: /vault/secrets/ |   +-------------------------+     |
|  +-----------+------------+                                   |
|              | HTTPS :8200 (K8s JWT + TLS)                   |
+--------------|------------------------------------------------+
               v
+---------------------------------------------------------------+
| Vault HA Cluster (Raft Integrated Storage, RHEL 9 / Rocky 9) |
|  +--------------------+  Raft :8201  +--------------------+   |
|  | vault-01 (Leader)  |<============>| vault-02 (Follower)|   |
|  | 10.10.1.11:8200    |              | 10.10.1.12:8200    |   |
|  +---------+----------+              +---------+----------+   |
|            ^              Raft :8201              ^           |
|            +---------------+-------------------+             |
|                            v                             |
|                     +--------------------+                |
|                     | vault-03 (Follower)|                |
|                     | 10.10.1.13:8200    |                |
|                     +---------+----------+                |
|                              |                              |
|         +------------------+------------------+             |
|         v                  v                  v             |
| [Auto-Unseal KMS :443]          [PostgreSQL 16 :5432]      |
| (Transit / Cloud KMS)           (Dynamic Role Engine)      |
+---------------------------------------------------------------+

Architecture decision : Integrated Storage (Raft) embeds the state machine inside the Vault process, eliminating cross-component network hops and Consul maintenance overhead. A 3-node cluster tolerates one node failure and supports atomic hot backups via vault operator raft snapshot.

2.4 Injection Mode Comparison

The following comparison uses lists instead of a table to comply with allowed HTML tags.

Vault Agent Injector : Sidecar renders to tmpfs; memory-only (zero disk); supports auto-renewal and template hot-reload; resource cost: +1 Sidecar per Pod (~50MB).

Vault CSI Provider : DaemonSet mounts memory volume (no sidecar); memory-only (CSI ephemeral memory volume); static volume refresh only, no dynamic renewal; node-level DaemonSet, cheaper at scale.

External Secrets Operator (ESO) : Syncs to native K8s Secret; not memory-only by default (writes to etcd, needs KMS); scheduled poll & overwrite Secret; minimal ops, fits legacy env-only apps.

Production recommendation : For high-security financial/payment services needing dynamic DB credentials and zero-disk, use Vault Agent Injector . For 1000+ microservice Pods consuming only static KV v2, use External Secrets Operator (ESO) with cluster-level NetworkPolicy restricting egress.

3. Production Baseline & OS Hardening

3.1 Verified Baseline Specs

OS/Kernel : Rocky Linux 9.4 (Kernel 5.14.0-427) / Ubuntu 24.04 LTS; swap disabled, XFS/ext4 with noatime.

Vault Version : HashiCorp Vault v2.2.1 (Community/Enterprise); includes 2.2.x policy path case-sensitivity & root alias hardening.

Node Hardware : 3 nodes, 8 vCPU / 16 GiB RAM / 200 GiB NVMe each; Raft WAL & BoltDB extremely sensitive to disk fsync latency.

Network/Ports : 10.10.1.11-13, API 8200/tcp, Raft 8201/tcp; full TLS 1.3, no plaintext over public net.

Secrets File : /etc/sysconfig/vault.env (perm 0600, owner vault); never hardcode KMS Key or Token in config/scripts.

3.2 OS Kernel Tuning (Step-by-Step)

Step 1: Disable Swap Permanently

swapoff -a
sed -ri '//swap/s/^#?/#/' /etc/fstab
free -h

Expected: Swap: 0B total.

Step 2: /etc/sysctl.d/99-vault-prod.conf

# Memory & anti-swap
vm.swappiness = 0
vm.overcommit_memory = 0
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

# TCP queues & port reuse
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

# Network buffers & file handles
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
fs.file-max = 2097152

Apply: sysctl --system and verify key params.

Step 3: Systemd Unit with mlock Capability

[Unit]
Description="HashiCorp Vault - Production Secrets Management"
Documentation=https://developer.hashicorp.com/vault/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
EnvironmentFile=/etc/sysconfig/vault.env
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=1048576
LimitMEMLOCK=infinity
LimitCORE=0

[Install]
WantedBy=multi-user.target

Key security mechanisms : LimitCORE=0 disables core dumps to prevent key extraction from /var/lib/systemd/coredump; CAP_IPC_LOCK + LimitMEMLOCK=infinity allow non-root vault user to call mlock() pinning physical memory pages.

3.3 Vault Raft Production vault.hcl (Annotated)

# /etc/vault.d/vault.hcl
ui = true
cluster_name = "prod-core-vault"
disable_mlock = false

api_addr = "https://10.10.1.11:8200"
cluster_addr = "https://10.10.1.11:8201"

# 1. API & Cluster Listener (enforce TLS 1.2+)
listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable = false
  tls_cert_file = "/etc/vault.d/tls/vault-node.crt"
  tls_key_file = "/etc/vault.d/tls/vault-node.key"
  tls_client_ca_file = "/etc/vault.d/tls/vault-ca.crt"
  tls_min_version = "tls12"
  tls_cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  telemetry {
    unauthenticated_metrics_access = false
  }
}

# 2. Integrated Storage (Raft)
storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault-01"
  retry_join {
    leader_api_addr = "https://10.10.1.11:8200"
    leader_ca_cert_file = "/etc/vault.d/tls/vault-ca.crt"
    leader_client_cert_file = "/etc/vault.d/tls/vault-node.crt"
    leader_client_key_file = "/etc/vault.d/tls/vault-node.key"
  }
  retry_join { ... vault-02 ... }
  retry_join { ... vault-03 ... }
  performance_multiplier = 1
  max_entry_size = 1048576
  autopilot_redundancy_zone = "az-a"
}

# 3. Auto-Unseal (Transit or Cloud KMS)
seal "transit" {
  address = "https://kms-transit.internal:8200"
  disable_renewal = "false"
  key_name = "autounseal-prod-core"
  mount_path = "transit/"
  tls_ca_cert = "/etc/vault.d/tls/kms-ca.crt"
}

# 4. Prometheus Telemetry
telemetry {
  prometheus_retention_time = "12h"
  disable_hostname = true
}

# 5. Lease TTL Baselines
default_lease_ttl = "1h"
max_lease_ttl = "24h"
log_level = "info"
log_format = "json"

3.4 Configuration "Four-Dimensional Breakdown"

① Must-Change Items (Per-Node Before Deploy)

api_addr

/ cluster_addr: Real routable internal IP (not 0.0.0.0 or 127.0.0.1), else Standby→Leader forwarding fails. storage.raft.node_id: Globally unique ( vault-01, vault-02, vault-03), immutable after init. storage.raft.path: Dedicated NVMe/SSD block device (e.g., /opt/vault/data), never shared with root FS or high-IO logs. seal block KMS address & key_name: Injected via controlled /etc/sysconfig/vault.env using VAULT_TOKEN (Transit) or cloud credentials; never in HCL plaintext.

② Coupled Items (Change One → Must Sync Other)

TLS & Metrics alignment : If

listener.tcp.telemetry.unauthenticated_metrics_access = false

, Prometheus must use a read -permission token to scrape /v1/sys/metrics?format=prometheus; prometheus_retention_time (recommend 12h) >> scrape_interval (e.g., 15s) to avoid metric gaps.

Lease TTL & App Refresh Window : default_lease_ttl ( 1h) couples with Vault Agent/ESO refreshInterval — refresh interval must be < 50% of TTL (recommend 20m), else DB auth fails after expiry.

K8s Webhook Timeout & Vault Concurrency :

vault-agent-injector
failurePolicy

& timeout (default 10s) must align with Vault

8200
somaxconn

and downstream DB connection latency.

③ Performance Tuning (Quantitative Formulas)

Raft Heartbeat Multiplier ( performance_multiplier ) : heartbeat_interval = 1000ms × multiplier, election_timeout = 1000ms × multiplier. Same-region dual-DC (RTT < 2ms) + NVMe ( fsync P99 < 2ms) → 1. Cross-AZ (RTT 5–15ms) or cloud disks (EBS gp3/ESSD PL1) → 2 or 3 to prevent flapping.

Vault Node Memory & LimitNOFILE :

Recommended GiB = 2 (base) + (active_leases/100k) × 1.5 + 2 (Raft cache)

. E.g., 400k leases → 10 GiB min (baseline 16 GiB).

LimitNOFILE = concurrent_clients × 2 + raft_peers × 16 + 4096

; set to 1048576 to avoid exhaustion.

PostgreSQL/MySQL Dynamic Credential Pool :

max_connections ≥ max_Pods × per_Pod_pool + (Vault_nodes × 10) + 20 (admin)

.

④ Absolute Red Lines (Touch = Irreversible Failure)

disable_mlock = true

on bare metal/VM: exposes keys to OS page-out risk. Only allowed in containers lacking CAP_IPC_LOCK with host swap already disabled. tls_disable = true in production: plaintext HTTP leaks Root Token, K8s JWT, DB passwords on cluster network.

Arbitrarily increasing max_entry_size (default 1MB): >4MB KV writes block Raft log replication, causing Follower sync timeouts.

Changing seal type in-place: Shamir→Auto-Unseal migration requires vault operator unseal -migrate; editing vault.hcl and restarting leaves cluster unsealable.

3.5 K8s Auth & Dynamic DB Secrets Init

After unseal, enable kubernetes auth and database engine with least-privilege policies:

vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://10.20.0.1:443" \
  kubernetes_ca_cert=@/etc/vault.d/tls/k8s-ca.crt \
  disable_local_ca_jwt="true"

vault secrets enable database
vault write database/config/prod-pg-orders \
  plugin_name="postgresql-database-plugin" \
  allowed_roles="orders-rw-role" \
  connection_url="postgresql://{{username}}:{{password}}@10.10.2.21:5432/orders?sslmode=require" \
  username="vault_admin" \
  password="${PG_VAULT_ADMIN_PASS}" \
  password_authentication="scram-sha-256"

vault write database/roles/orders-rw-role \
  db_name="prod-pg-orders" \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REASSIGN OWNED BY \"{{name}}\" TO vault_admin; \
    DROP OWNED BY \"{{name}}\"; \
    DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

App policy /etc/vault.d/policies/prod-orders-policy.hcl (Vault 2.2.x fixes path case-sensitivity & root alias bypass):

path "database/creds/orders-rw-role" {
  capabilities = ["read"]
}
path "secret/data/prod/orders/*" {
  capabilities = ["read"]
}
path "secret/metadata/prod/orders/*" {
  capabilities = ["list"]
}

Bind to K8s prod-pay namespace orders-svc ServiceAccount; deploy Deployment with Vault Agent Injector annotations rendering db.env to tmpfs.

3.6 Disk Scheduler, Mount & limits.d

Raft BoltDB requires fsync() per transaction. On NVMe, disable atime and use none I/O scheduler:

cat /sys/block/nvme0n1/queue/scheduler  # expect [none]
grep "/opt/vault/data" /etc/fstab || echo "/dev/nvme0n1p1 /opt/vault/data xfs defaults,noatime,nodiratime 0 0"
/etc/security/limits.d/99-vault.conf

:

vault  soft  nofile  1048576
vault  hard  nofile  1048576
vault  soft  memlock unlimited
vault  hard  memlock unlimited
vault  soft  core    0
vault  hard  core    0

3.7 Audit Device with Anti-Fillup

Vault refuses all requests if audit write fails. Enable file audit with HMAC-SHA256 masking, and logrotate using postrotate + SIGHUP (not copytruncate):

vault audit enable file file_path=/var/log/vault/audit.log log_raw=false
# /etc/logrotate.d/vault
/var/log/vault/audit.log {
  daily
  rotate 30
  compress
  delaycompress
  missingok
  notifempty
  create 0640 vault vault
  postrotate
    /bin/kill -HUP $(systemctl show --property MainPID --value vault) 2>/dev/null || true
  endscript
}

3.8 ESO + NetworkPolicy for Static KV Consumers

For large stateless clusters reading only static KV v2, ESO avoids per-Pod sidecar overhead. Example SecretStore, ExternalSecret, and egress-restricting NetworkPolicy allowing only Vault IPs (10.10.1.11/12/13:8200) and DNS (53).

3.9 KV v2 Version Governance

Prevent CI/CD version bloat:

vault secrets enable -path=secret kv-v2
vault write secret/config max_versions=10 cas_required=true
vault kv put secret/prod/orders/payment stripe_api_key=@/tmp/stripe_key.txt webhook_secret=@/tmp/webhook_sec.txt
vault kv metadata get secret/prod/orders/payment

Metadata shows max_versions=10, cas_required=true (optimistic locking).

3.10 Internal PKI for mTLS

Short-lived service certs (72h TTL) via PKI engine:

vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=8760h pki_int
vault write pki_int/roles/prod-pay-mtls allowed_domains="prod-pay.svc.cluster.local" allow_subdomains=true max_ttl="72h" key_type="ec" key_bits=256
vault write -format=json pki_int/issue/prod-pay-mtls common_name="orders.prod-pay.svc.cluster.local" ttl="24h"

Result: 24h cert, auto-expires in 1 day even if leaked.

4. Troubleshooting One-Liners & PromQL Alerts

4.1 Key One-Liners

10s Health Check : Loop over nodes, curl /v1/sys/health, parse sealed, standby, version. sealed=true = P0; all standby=true = quorum loss.

Active Lease Count :

vault list -format=json sys/leases/lookup/database/creds/orders-rw-role | jq -r 'length'

. Normal ≈ Pod replicas × (1–2). >10× indicates leak (app re-requests per query).

Audit 403 Top Paths :

tail -n 2000 /var/log/vault/audit.log | jq -r 'select(.type=="response" and .error != null) | "\(.time) \(.request.remote_address) \(.request.path) \(.error)"' | sort | uniq -c | sort -nr | head -n 10

.

Raft Disk fsync & DB Size :

ls -lh /opt/vault/data/raft/raft.db && iostat -xz 1 3 | awk '/nvme0n1|Device/ {print $1, $4, $10, $11}'

. w_await > 20ms triggers Raft instability.

Pod tmpfs Verification :

kubectl -n prod-pay exec deploy/orders-service -c orders-api -- df -hT /vault/secrets && ls -la /vault/secrets

. Must show Type=tmpfs, perms 0640.

Token Bloat :

vault list -format=json auth/token/accessors | jq -r 'length'

. Normal < few thousand; >50k monotonic increase → audit auth/token/create callers.

TLS Handshake Breakdown :

curl -sk --cacert ... -w "dns=%{time_namelookup}s tcp=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s" -o /dev/null "https://10.10.1.11:8200/v1/sys/health"

.

K8s TokenReview Connectivity : vault read auth/kubernetes/config — verify disable_local_ca_jwt=true, correct kubernetes_host.

Cert Expiry Check :

openssl s_client -connect 10.10.1.11:8200 -servername 10.10.1.11 </dev/null 2>/dev/null | openssl x509 -noout -dates -subject

. Alert if <15 days.

4.2 Core PromQL Alerts

# 1. Node sealed or down
(vault_core_unsealed == 0) or (up{job="vault-prod"} == 0)

# 2. Leader loss or split-brain
sum(vault_core_active{job="vault-prod"}) != 1

# 3. Lease count runaway
vault_expire_num_leases{job="vault-prod"} > 50000

# 4. P99 latency > 500ms for 3m
histogram_quantile(0.99, sum(rate(vault_core_handle_request_bucket{job="vault-prod"}[5m])) by (le)) > 500

PostgreSQL orphan role check (run on replica):

SELECT usename, valuntil FROM pg_user WHERE usename LIKE 'v-kubernet-%' AND valuntil < NOW() - INTERVAL '1 hour' ORDER BY valuntil ASC LIMIT 20;

5. Three Real Production Incident Postmortems

Incident 1: Mass Pod Rollout → Vault Auth Avalanche & DB Account Exhaustion

Symptoms

vault_core_handle_request

P99: 18ms → 4,200ms.

Pods stuck in Init:0/1 or CrashLoopBackOff with 500 Internal Server Error.

PostgreSQL: FATAL: sorry, too many clients already.

Root Cause Chain

App has idle in transaction long transactions holding table locks.

Vault creation_statements used broad GRANT ... ON ALL TABLES IN SCHEMA public. Each new Pod's dynamic cred request tried to grant on all tables, blocking behind the long transaction.

Pod timeouts triggered K8s retry storm, piling hundreds of blocked sessions, saturating PG max_connections.

Fix & Quantitative Result

Pre-create parent role role_orders_rw with ALTER DEFAULT PRIVILEGES; change Vault creation_statements to lightweight GRANT role_orders_rw TO "{{name}}" (no table scan/lock).

Dynamic cred issuance P99 : Before 4,200 ms (timeouts) → After 14 ms (-99.6%).

PG vault_admin lock waits : Before 45+ concurrent blocks → After 0 (eliminated).

180 Pod scale-up ready time : Before >8 min, partial failures → After 28 sec all ready (17× faster).

Rollback:

GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO role_orders_rw;

instantly, no Vault/Pod restart.

Incident 2: Raft Split-Brain & Frequent Leader Flapping

Symptoms

vault-agent

errors: context deadline exceeded on lease renew.

Prometheus alert: sum(vault_core_active) != 1.

Root Cause

Cross-DC RTT 6–8ms; nightly vault operator raft snapshot save at 02:14 coincided with 1.8GB raft.db flush. performance_multiplier=1 (default) → heartbeat timeout ~1000ms. Disk fsync spike + RTT jitter caused Leader to miss quorum ACKs, triggering step-down & re-election.

Fix & Result

Set performance_multiplier=3 on all 3 nodes (widens tolerance to ~3s).

Clean stale leases & enforce KV max_versions=10, shrinking raft.db from 1.8GB → 180MB.

Rolling restart: Followers first, then vault operator step-down Leader.

Unexpected leader changes/day : Before 3–5 → After 0 (30 days stable).

Snapshot backup API P99 : Before 1,450 ms (with 503s) → After 32 ms (zero errors).

Raft disk w_await peak : Before 48 ms → After 2.1 ms.

Rollback: If real failover latency (1s→3s) unacceptable, set multiplier=2 and roll restart.

Incident 3: Node Power Loss → Stuck Sealed, Auto-Unseal 403

Symptoms

vault-03

host power failure; on reboot vault.service running but vault status shows Sealed: true.

Root Cause

Transit Auto-Unseal used a periodic token in /etc/sysconfig/vault.env without auto-renewal service. Token expired after ttl during long uptime; on restart, unseal failed with 403 bad token.

Fix

On KMS Transit cluster, issue dedicated Periodic Token with period="720h", no max_ttl (renew within period = never expires), policy autounseal-policy.

Ensure seal "transit" { disable_renewal = "false" } so Vault auto-renews its own token.

Update VAULT_TOKEN in /etc/sysconfig/vault.env (perm 0600), restart vault-03 → auto-unseal succeeds.

# On KMS Transit cluster
vault token create -policy="autounseal-policy" -period="720h" -orphan

Rollback: Use offline Recovery Keys (3/5 Shamir shares) via vault operator unseal to manually unseal, then fix Transit policy.

6. Production Ops Scripts, Risk Matrix & Engineering Iron Laws

6.1 Raft Hot Backup + Integrity Script (with --dry-run )

Strict set -euo pipefail, env-file isolation, pre-flight health check (must be unsealed + leader exists), snapshot save, SHA256 verification, retention cleanup. Run before any high-risk change or version upgrade.

6.2 10-Second Blackbox Collector ( vault-blackbox-collect.sh )

Captures cluster state, Prometheus metrics snapshot, last 1000 journal lines, OS resources (memory, sockets, iostat) into a tarball, auto-excluding sensitive files ( vault.env, raft.db).

6.3 Change Risk Matrix (Six Elements)

Modify vault.hcl storage/listener : Node restart → brief Standby reconnect. Pre-checks: Backup Raft snapshot; change 1 Follower first, verify, then roll Leader. Rollback: Restore backed-up vault.hcl + systemctl restart vault.

Major version upgrade (1.19→2.2) : Breaking policy validation (root alias ban). Pre-checks: Validate existing policy syntax; upgrade all Followers, then step-down Leader last. Rollback: Downgrade binary + vault operator raft snapshot restore -force.

Change DB role creation_statements : Affects only new leases. Pre-checks: Test SQL syntax & lock time in test Role before overwriting prod. Rollback: Re-run old vault write database/roles/... to overwrite.

Revoke lease prefix ( revoke-prefix ) : Batch kills matching DB connections → possible app blip. Pre-checks: vault list sys/leases/lookup/... confirm count; run in low-traffic window. Rollback: Irreversible; trigger Pod rollout to re-acquire creds.

6.4 Pre-Production 6-Step Security Baseline

No stray root tokens:

vault list -format=json auth/token/accessors | jq -r 'length'

.

Audit devices healthy: vault audit list -detailed.

No insecure listeners: grep -E "tls_disable|disable_mlock" /etc/vault.d/vault.hcl (both must be false).

Controlled env file perms:

stat -c "%a %U:%G %n" /etc/sysconfig/vault.env /etc/vault.d/vault.hcl

→ 600 root:root and 640 vault:vault.

Raft Autopilot tolerance ≥1:

vault operator raft autopilot state | grep -E "Healthy|Failure Tolerance"

.

Pod secret mount is tmpfs:

kubectl -n prod-pay exec deploy/orders-service -c orders-api -- df -hT /vault/secrets

.

6.5 Five Core Engineering Iron Laws

No static password reuse across environments : Code, Helm values, CI/CD vars only contain Vault path refs (e.g., secret/data/prod/app); zero plaintext secrets.

Dynamic over static, Memory-Only over disk : DB, MQ, Cloud IAM creds → dynamic engines with short TTL; containers use tmpfs mounts; hosts fully disable Swap & Core Dump.

Control-plane HA & Snapshot Routine : Odd-node Raft (3/5) across failure domains; daily automated raft snapshot with SHA256 to geo-redundant cold storage; 5-min full-state rebuild SLA.

Audit Log & Disk Pressure Coupled : At least two redundant audit sinks (or file + strict logrotate SIGHUP); disk usage >80% pre-alert to prevent audit-write-block-induced cluster-wide issuance outage.

Least Privilege & Short Leases : K8s auth roles bind exact bound_service_account_names + bound_service_account_namespaces (no * wildcard); dynamic DB default_ttl ≤ 1h; internal mTLS certs ≤ 72h.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

incident responseRaft ConsensusSecrets ManagementHashiCorp VaultAuto-UnsealDynamic SecretsKubernetes IntegrationProduction Hardening
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.