Kubernetes ConfigMap & Secret Daily Management: From Creation to Security Hardening

Why Configuration Management Deserves a Dedicated Guide

ConfigMap and Secret appear simple but cause numerous production incidents. Three real scenarios illustrate the risks: (1) A Java service migrated to ConfigMap; changing spring.datasource.url triggered CrashLoopBackOff because the Deployment used both file-mounted properties and envFrom.configMapRef with a mismatched key name ( spring.profile vs spring.profile.active). envFrom silently drops missing keys while env.valueFrom.configMapKeyRef blocks Pod creation with CreateContainerConfigError. (2) Secrets stored in etcd are only base64-encoded; anyone with etcd snapshots or certificates can read all passwords and cloud credentials via etcdctl get. (3) TLS certificate rotation via Secret volume mount appeared ineffective overnight due to subPath mounts (kubelet never updates them) and Nginx not auto-reloading.

Environment Assumptions & Version Constraints

Examples target Kubernetes 1.29–1.35 (primarily 1.33/1.34), containerd 1.7/2.x, kubectl within one minor version, Calico/Cilium CNI, ingress-nginx 1.10+, etcd 3.5.x. Key version milestones: Secret encryption GA in 1.13, Immutable ConfigMap/Secret GA in 1.21, KMS v2 GA in 1.29, encryption wildcards ( *. / *.*) in 1.27, automatic encryption config reload GA in 1.29. immutable reduces apiserver/kubelet watch overhead, not read latency.

Deciding Between ConfigMap and Secret

Decision flow: (1) Can leakage cause direct damage? → Secret or external secrets. (2) Need audit of who read it? → Secret with audit policy + minimal RBAC. (3) Acceptable in Git plaintext? → ConfigMap; else Secret + encryption (SOPS/Sealed Secrets/external). Common items mapping:

DB host/port → ConfigMap

DB password → Secret

JVM params/log levels → ConfigMap

TLS cert/key → Secret ( kubernetes.io/tls)

Cloud AK/SK → Secret + external

Nginx upstreams → ConfigMap

Webhook signing key → Secret

Feature flags → ConfigMap

Image pull credentials → Secret ( kubernetes.io/dockerconfigjson)

JWT public key → ConfigMap

Never bundle private and public keys in one Secret for all services.

Anti-pattern: Stuffing Entire application.yml into One ConfigMap Key

Problems: diff shows whole multi-line string; placeholder ${DB_PASSWORD} relies on Spring resolution and env var existence; 1MiB size limit hits fast. Better: "config sharding" — separate keys for log-level, datasource-url, pool-size, keep only structural files like logback-spring.xml as multi-line values.

Secret Type Selection

Don't default to Opaque. Built-in types enforce required keys: kubernetes.io/tls needs tls.crt / tls.key (used by ingress-nginx, cert-manager, Istio) kubernetes.io/dockerconfigjson needs .dockerconfigjson (kubelet) kubernetes.io/basic-auth needs username /

password

kubernetes.io/ssh-auth

needs ssh-privatekey API server only validates key presence, not cert/key matching — must verify manually with openssl x509 -noout -pubkey and openssl pkey -pubout comparing SHA256.

Creation: From Imperative to Declarative

Imperative Commands for Quick Conversion

kubectl create configmap

/ secret with --from-literal, --from-file, --from-file=key=path, --from-file=dir/ (non-recursive, skips dotfiles, follows symlinks). Verify keys with kubectl get cm ... -o jsonpath='{.data}' | jq -r 'keys[]'. For Secrets, avoid plaintext in shell history: use read -s into a 600-mode file, kubectl create secret ... --from-file=key=file, then shred -u. Or write YAML with stringData placeholder, sed replace (risky with special chars), chmod 600, apply, delete.

Dry-run to Declarative YAML

kubectl create configmap ... --dry-run=client -o yaml > file.yaml

(local render) vs --dry-run=server (server validation including admission webhooks). Use server mode when policies exist, version compatibility uncertain, or using generateName. kubectl diff -f file.yaml exit codes: 0=no diff, 1=diff, >1=error; usable in CI to skip rollouts.

Declarative YAML Best Practices

ConfigMap: use | (keep trailing newline) vs |- (strip) for multi-line; quote numbers ( "8080") because YAML parses unquoted as integer but API requires string; keys must match [-._a-zA-Z0-9]; data and binaryData keys cannot overlap; binaryData values must be standard base64 ( base64 -w0). Secret: prefer stringData for editing (auto base64 to data); always quote values (single quotes prevent $ expansion); immutable: true must be set at creation (cannot change later).

Naming & Labeling Conventions

Pattern: <app>-<purpose>[-<version>] (e.g., order-service-config, order-service-secret, order-service-config-v3). Mandatory labels: app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/managed-by (e.g., argocd). Enables finding manually created objects: kubectl get cm -A -l 'app.kubernetes.io/managed-by!=argocd'.

Consumption: Env Vars vs File Mounts

Four Consumption Methods Behavior Matrix

Comparison of four methods:

env.valueFrom.configMapKeyRef : Reads at container creation; no auto-update; missing key → Pod stuck in CreateContainerConfigError; suitable for few stable scalar configs.

envFrom.configMapRef : Reads at container creation; no auto-update; missing key → missing var, Pod starts normally; suitable for batch of homogeneous configs.

volumeMounts (directory) : Reads at container creation; yes (eventual, latency); missing key → Pod stuck in ContainerCreating; suitable for config files.

volumeMounts + subPath : Reads at container creation; no auto-update; missing key → Pod stuck in ContainerCreating; suitable for single file override, immutable dir.

Missing-key behavior is default ( optional unset). Adding optional: true makes env.valueFrom behave like envFrom (silent missing).

Env Var Pitfalls

(1) Updates never propagate — env vars are process initial state; must restart Pod. (2) Inconsistent failure: env.valueFrom.secretKeyRef with wrong key blocks Pod; same missing key via envFrom silently omits var, app fails later with default or crash. Debug with kubectl get rs -l app=... -o json | jq ... to compare env vs envFrom across ReplicaSets.

Directory Volume Mount (Recommended)

Use volumes.configMap.items[] to select specific keys, set defaultMode and per-item mode. Without items, all keys become files, mixing scalars and binaries. path can include subdirectories (e.g., ssh/id_rsa) with custom mode: 0400.

subPath Limitation

kubelet never updates subPath mounts (bind-mounts single file, symlink swap doesn't penetrate). Any config change requires Pod rebuild. Alternatives: mount to separate dir and cp / ln -s in startup script; init container merges image defaults with ConfigMap into emptyDir; accept full directory overwrite and put all needed files in ConfigMap.

Non-root Permission Issues

Default defaultMode: 0644, owner root:root. Non-root containers get EACCES. Fix: set defaultMode: 0440 + fsGroup in pod securityContext (kubelet chowns to fsGroup and adds group bits: 0644→0674, 0440→0470). For private keys, use 0400 without fsGroup and run as root or matching UID. fsGroupChangePolicy: OnRootMismatch (1.23 GA) avoids recursive chown on every start.

Secret Volume tmpfs Characteristics

Secret volumes are tmpfs (memory-only). 1000 Pods × 4KB = 4MB negligible; but 1MB cert chain × 1000 Pods = 1GB memory (visible in df -h -t tmpfs under /var/lib/kubelet/pods/.../volumes/kubernetes.io~secret). Node restart requires re-pull from apiserver; if apiserver down, Pods stall at mount.

Selection Guide

DB connection strings, ports → env + secretKeyRef / configMapKeyRef (explicit, missing key fails fast)

Batch of feature flags → envFrom (concise, but silent missing — add startup validation)

Nginx/Prometheus/Redis config files → Volume directory mount (supports updates, matches app file-read behavior)

Single file override, restart acceptable → Volume + subPath (only when directory structure immutable)

Private keys, certs → Volume mount + mode: 0400 (avoids /proc/<pid>/environ leakage)

Hot-reload certs → Volume directory (no subPath) (enables app reload)

Changes: How to Update & Make Effective

Two Independent Propagation Paths

Path A (volume mounts only): kubectl apply → apiserver → etcd → watch → kubelet syncLoop → update ..data symlinks → container files update (30s–2min latency). Path B (env vars, subPath): only Pod rebuild works. Rebuild methods: kubectl rollout restart deploy/name (adds restartedAt annotation, rolling update), kubectl delete pod (single, for debug), template change (implicit, not recommended), kubectl scale to 0 (full outage, forbidden).

Rollout Restart Details

kubectl rollout status deploy/name --timeout=300s

shows progress. Stalls due to: new Pod ImagePullBackOff / CrashLoopBackOff / CreateContainerConfigError; readiness probe failures; resource exhaustion; small maxUnavailable + slow startup. PDBs constrain evictions ( kubectl drain), not Deployment rolling updates (direct Pod deletion).

Canary: Test Config on One Replica

Create separate Deployment order-service-canary (replicas:1, track: canary label) referencing new ConfigMap order-service-config-v2. Service selector app.kubernetes.io/name=order-service picks both stable and canary. Monitor canary logs with

kubectl logs -l 'app.kubernetes.io/name=order-service,track=canary'

. Canary must have independent alert thresholds (absolute counts, not rates) and fixed replicas ( minReplicas=maxReplicas=1) to prevent HPA scaling it up.

Immutable ConfigMap Rolling Release

With immutable: true, update = create new (suffix with git short hash/version) + switch reference. Steps: copy YAML, change name, set immutable: true, kubectl apply --dry-run=server, apply, then

kubectl patch deployment ... --type=json -p='[{"op":"test","path":"/spec/template/spec/volumes/0/name","value":"config"},{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"order-service-config-v3"}]'

. test op prevents wrong index patch. Better: use Kustomize configMapGenerator for automatic hash suffix and reference update.

Kustomize: Automatic Hash Alignment

configMapGenerator

with files / literals, secretGenerator with envs (from secrets/prod.env injected by CI). generatorOptions.disableNameSuffixHash: false (default) appends content hash to name and updates all references in Deployment ( envFrom, env.valueFrom, volumes.configMap.name), triggering automatic rollout on content change.

secretGenerator

envs

does variable substitution ( $VAR); passwords with $ break it — use files instead (raw content, only strips trailing newline).

Reloader: Auto-restart on Config Change

Install stakater/reloader Helm chart ( watchGlobally=false, reloadOnCreate=false). Annotate workload: reloader.stakater.com/auto: "true" (any referenced CM/Secret change triggers rollout) or reloader.stakater.com/search: "true" (only CM/Secret with reloader.stakater.com/match: "true"). Risks: no content validation (syntax errors roll out to all Pods), no debounce (multiple CI commits = multiple rollouts), broad RBAC (watch all namespaces).

Application-level Hot Reload (Most Controlled)

Nginx: sidecar watches /etc/nginx/conf.d and /etc/nginx/certs with find ... -exec sha256sum | sort | sha256sum; on change, run nginx -t then nginx -s reload. Prometheus: --web.enable-lifecycle + wget -qO- --post-data='' http://localhost:9090/-/reload (operator uses config-reloader sidecar). Spring Boot Actuator: curl -X POST /actuator/refresh (only rebinds @ConfigurationProperties / @RefreshScope beans, not datasource/port).

Pre-change Checklist

Backup current objects (especially Secrets): kubectl get cm,secret -n ns -o yaml > backup/...raw.yaml, strip resourceVersion, uid, creationTimestamp, generation, managedFields for re-apply.

Diff before/after:

kubectl get cm name -o yaml > before.yaml; kubectl apply -f ...; kubectl get cm name -o yaml > after.yaml; diff ...

Test in non-prod first.

Troubleshooting: Missing Keys, Failed Pods, Permission Issues

Troubleshooting Flow

Pod won't start: CrashLoopBackOff → container logs (app read wrong/missing config); CreateContainerConfigError → describe Events: missing object/key; ContainerCreating long → describe Events: mount failure → node journalctl -u kubelet. Pod up but wrong: config propagated but old Pods still serving; config not propagated ( subPath /env var/latency); partial (old/new ReplicaSets). Permission: Forbidden (RBAC); file EACCES ( defaultMode / fsGroup vs runAsUser); SELinux/AppArmor.

Key Missing ( CreateContainerConfigError )

Event:

couldn't find key DB_PASSWORD in Secret prod/order-service-secret

. kubelet retries ( x5 over 58s), auto-recovers when key added. Verify actual keys:

kubectl get secret ... -o jsonpath='{.data}' | jq -r 'keys[]'

. Causes: namespace mismatch (Deployment in wrong ns), invisible chars in key name (check with od -c).

Object Missing

Event: configmap "order-service-config" not found. Check existence and events. optional: true on envFrom makes missing object silent — Pod starts with zero env vars, app fails later (classic "service up but can't connect DB"). Audit cluster for optional: true usage on critical configs.

Mount Failure ( ContainerCreating >2min)

Common Events: object not found (create it); permission denied (node kubelet/SELinux — check journalctl -u kubelet); timeout waiting (apiserver unreachable/kubelet stuck); failed to fetch (network/cert); secret type mismatch (check type). Increase --event-ttl to 24h.

Config Changed But Not Effective

Path: confirm object changed → check container file ( ls -la /mount/path shows symlinks to ..data/ timestamp dirs). Timestamp unchanged = kubelet not synced yet (wait 2min). Timestamp changed but app sees old = app cached at startup (needs reload/restart). subPath non-empty = root cause. Env var = never updates, must rebuild Pod.

Partial Pods Effective

(1) kubelet cross-node fan-out latency (tens of seconds normal). Verify with per-pod stat -c "%y" /path/file. (2) Old/new ReplicaSets reference different ConfigMaps — check

kubectl get rs -l app=... -o custom-columns=NAME:.metadata.name,CM:.spec.template.spec.volumes[0].configMap.name

. (3) Helm checksum/config annotation stale: manual kubectl apply bypasses Helm, or .Files.Get external file not in checksum. Compare annotation value vs actual ConfigMap data hash.

RBAC Issues

kubectl auth can-i get secrets -n prod --as=system:serviceaccount:prod:order-svc

checks authorization (not admission webhooks). App reading Secret via API needs Role with resourceNames limiting to specific Secrets (without it, compromised Pod reads all namespace Secrets). resourceNames disables list / watch — acceptable tradeoff.

File Permission Errors Despite Correct ls -l

Causes: parent directory inaccessible (e.g., /etc 0700); fsGroup adds group bits (0400→0470) but container UID not in group; SELinux/AppArmor denial ( ausearch -m avc, dmesg | grep avc). Fix: proper seLinuxOptions, not permissive mode.

Full Debugging Case Study

02:00, 6/20 Pods CrashLoopBackOff after ConfigMap change 2hrs prior. Logs: Could not resolve placeholder 'REDIS_TIMEOUT_MS'. ConfigMap key: redis_timeout_ms (lowercase), app expects REDIS_TIMEOUT_MS. envFrom doesn't auto-uppercase. Failed Pods on new nodes (node-11/12) with cached old image: old image read application.yml with default ${REDIS_TIMEOUT_MS:500}; new image used env var, no default → crash. Fix: patch ConfigMap adding uppercase key REDIS_TIMEOUT_MS: "300" without removing lowercase. Lessons: enforce key naming convention (CI regex ^[A-Z][A-Z0-9_]*$), ensure config changes compatible with all running image versions, unify default-value strategy (all or none).

Governance: Orphan Configs, Expired Secrets, Capacity

Finding Orphan ConfigMaps/Secrets

Script scans all workloads (Deploy/STS/DS/CronJob/Job) for references in envFrom, env.valueFrom, volumes.configMap/secret, projected, imagePullSecrets, plus Ingress TLS Secrets. Outputs unreferenced objects (excluding system namespaces, ownerReferences). Limitations: misses indirect references, operator-managed objects, dynamic bindings. Process: report → business confirm → mark → re-scan → backup YAML → delete → observe 7 days → cleanup backup.

Identifying Expired Secrets

(1) Long-unrotated Secrets:

kubectl get secret -A -o json | jq -r '.items[] | select(.type != "kubernetes.io/service-account-token") | select((.metadata.ownerReferences // []) | length == 0) | [.metadata.namespace, .metadata.name, .type, .metadata.creationTimestamp] | @tsv' | sort -k4 | head -40

. Check managedFields for last modification. (2) TLS cert expiry: script decodes tls.crt, runs openssl x509 -noout -enddate, computes days remaining, sorts ascending. (3) Registry creds: decode .dockerconfigjson, extract server/username, test with skopeo list-tags --authfile.

Capacity Governance

Single ConfigMap/Secret limit 1MiB. Near-limit detection:

kubectl get cm,secret -A -o json | jq ... select($t > 200000)

(approximate). Exceeding limit fails create and can stall Deployment rollout (referenced object missing). Prevent in CI with --dry-run=server. Etcd total quota (default 2GB, often 8GB) — millions of stale Secrets cause etcdserver: mvcc: database space exceeded (cluster read-only). Recovery: etcdctl alarm disarm, compact, defrag per member sequentially. Prevention: monitor

etcd_db_total_size_in_bytes / etcd_server_quota_backend_bytes

>60% alert; namespace ResourceQuota with count/configmaps, count/secrets (set to 1.5× current).

Daily Health Check CronJob

Script checks: non-Running Pods with config errors; last 24h FailedMount events; config missing key errors; workloads using subPath; optional: true references; objects >200KB; workloads injecting Secrets via env vars. Runs via CronJob with config-auditor ClusterRole (get/list cm,secret,pods,events,namespaces,deploy,sts,ds,jobs,cronjobs,ingresses). Warning : secrets list = read all Secrets cluster-wide. Mitigation: split checks (in-cluster no secret content + external with controlled kubeconfig) or run from bastion.

Security Hardening: Making Secrets Real Secrets

Default State: Unencrypted, Readable

Secret in etcd = base64 only. Anyone with get secrets sees plaintext ( base64 -d). Etcd snapshot/backup = all Secrets. Env var injection exposes to /proc/<pid>/environ (all processes in container, shared PID namespace, CAP_SYS_PTRACE /privileged). Private keys must use volume mounts, not env vars.

Encryption at Rest: From Zero

Target: etcd stores ciphertext. Assumption: kubeadm, 3 control-plane, stacked etcd. Check: apiserver --encryption-provider-config present? Test: create Secret, etcdctl get /registry/secrets/ns/name | hexdump -C. Plaintext shows k8s\x00...Secret; encrypted shows k8s:enc:aescbc:v1:key-202611:....

Steps: (1) Each control-plane node: head -c 32 /dev/urandom | base64 (44 chars, 32 bytes) — safeguard this key (loss = all Secrets lost). (2) Write /etc/kubernetes/enc/enc-config.yaml identical on all nodes: resources: [secrets, configmaps],

providers: [{aescbc: {keys: [{name: key-202611, secret: "${AES_KEY}"}]}, {identity: {}}]

. Order matters: first provider encrypts writes, reads try each in order; identity last allows reading existing plaintext. aescbc preferred over aesgcm (200k write rotation required, no auto-rotate) and secretbox (newer, compliance issues). File chmod 600 root:root. (3) Edit static Pod /etc/kubernetes/manifests/kube-apiserver.yaml: add

--encryption-provider-config=/etc/kubernetes/enc/enc-config.yaml

, --encryption-provider-config-automatic-reload=true (1.29 GA, 1.26-1.28 alpha), mount hostPath volume /etc/kubernetes/enc. kubelet recreates apiserver Pod per node sequentially; wait for kubectl get --raw='/readyz?verbose' showing readyz check passed. (4) Verify: recreate test Secret, check etcd prefix k8s:enc:. (5) Bulk re-encrypt existing: per-namespace kubectl get secrets -n ns -o json | kubectl replace -f - (throttle with sleep). (6) Confirm all encrypted: scan all Secrets via etcdctl for k8s:enc: prefix. (7) Remove identity: {} after full encryption (prevents plaintext writes). Rollback: prepend identity: {} to providers list, reload.

Key rotation: add new key to second position → wait reload → move new key to first → wait reload → bulk rewrite → confirm new key prefix on all nodes → remove old key. Multi-node consistency: sha256sum /etc/kubernetes/enc/enc-config.yaml across nodes; monitor

apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds

metric.

KMS v2: Keys Never Leave Control Plane

Static encryption stores key on apiserver nodes — node compromise = key + ciphertext. KMS v2 uses external KMS (cloud KMS, Vault Transit) via gRPC plugin. Config:

providers: [{kms: {apiVersion: v2, name: vault-kms, endpoint: unix:///var/run/kmsplugin/socket.sock, timeout: 3s}}, {identity: {}}]

. apiVersion: v2 distinguishes from deprecated v1. Plugin deployed as static Pod per control-plane node, socket at configured path. Validation: KMS HA, ms latency; socket exists srw-rw----; plugin health via Status RPC (check apiserver logs); metrics

apiserver_encryption_config_controller_kms_operations_latency_seconds

P99 < 100ms. Critical: KMS unavailability = apiserver cannot decrypt = cluster unusable. Must have local cache/fallback, tested rollback plan, strict KMS availability alerts. Fallback config: keep aescbc provider alongside KMS (read tries both, write prefers KMS).

RBAC Least Privilege

(1) No wildcard resources: ["*"] + verbs: ["*"] on Secrets. Audit with jq. (2) Developers read-only, no Secrets: ClusterRole with pods, pods/log, services, endpoints, nodes, events, configmaps, deployments, etc. — simply omit secrets rule. (3) Apps mount Secrets via kubelet credentials ( system:node:<node>), not ServiceAccount — RBAC cannot restrict mounted Secrets; only prevention is not mounting them. (4) Audit who reads Secrets: audit.k8s.io/v1 policy with level: Metadata for secrets get / list (records who/when/what, not content). level: Request logs request body = Secret plaintext in audit log — anti-pattern. Query audit log for list secrets by ServiceAccounts (normal apps don't list).

Admission Control: Shift Rules Left

Kyverno policies: (1) Block sensitive keys in ConfigMap (Audit first): JMESPath

keys(@)[?contains(@, 'PASSWORD') || contains(@, 'PASSWD') || contains(@, 'SECRET') || contains(@, 'API_KEY') || contains(@, 'PRIVATE_KEY')] | length(@) > 0

. Excludes TOKEN to avoid false positives on TOKEN_TTL_SECONDS. (2) Require labels on Secret:

pattern: metadata.labels: {app.kubernetes.io/name: "?*", owner: "?*"}

. (3) Disallow long-lived ServiceAccount token Secrets (Enforce): deny type: kubernetes.io/service-account-token (except kube-system). Use kubectl create token sa -n ns --duration=2h instead.

External Secrets Management: No Passwords in Secret

(1) External Secrets Operator (ESO): SecretStore (Vault backend) + ExternalSecret mapping remote keys to K8s Secret keys, refreshInterval: 1h, creationPolicy: Owner (deletes Secret when ExternalSecret deleted). Still writes K8s Secret — pair with encryption at rest. (2) Secrets Store CSI Driver: SecretProviderClass mounts directly from Vault via CSI, no etcd copy if secretObjects omitted. Tradeoffs: DaemonSet on every node, mount failure blocks Pod start, rotation depends on driver poll interval. (3) Sealed Secrets: kubeseal encrypts Secret to SealedSecret (namespace+name bound by default), controller decrypts to Secret. Solves Git plaintext only. Controller key rotation: backup sealed-secrets-key* Secret, add new key → re-encrypt all → delete old key.

Selection: Vault/cloud KMS → ESO (simpler, smaller blast radius); compliance "no etcd landing" → CSI Driver without secretObjects; only Git plaintext concern → Sealed Secrets/SOPS; small team, no external KMS → encrypted Secret + GitOps + Sealed Secrets.

Engineering: Codifying the Process

Directory Structure & Config Ownership

Config and code same repo, same PR. deploy/base/ (deployment, service, config files), deploy/overlays/{staging,prod}/ with kustomization.yaml, configmap-patch.yaml (only differing keys), secrets/{staging,prod}.env (not in Git, CI injects). Kustomize merges ConfigMap patches (key-wise override, doesn't delete omitted keys) — verify with

kubectl kustomize overlays/prod | awk '/^kind: ConfigMap/{f=1} f{print} /^---$/{f=0}'

.

CI Validation: Catch Errors Pre-merge

Script ci-validate.sh runs: (1) kubectl kustomize build; (2) kubeconform schema validation (strict, ignore missing schemas, target K8s version); (3) kubectl apply --dry-run=server (if cluster access); (4) Python scan rendered YAML for ConfigMap keys matching (password|passwd|secret|token|api[_-]?key|private[_-]?key) with non-placeholder values — block (misreport TOKEN_TTL_SECONDS acceptable, low fix cost vs high leak cost); (5) All Secrets must have owner label (phase: only new/modified Secrets via git diff); (6) Warn on subPath usage.

GitOps Gotchas

Argo CD: (1) prune: true deletes dynamically created Secrets (ESO, cert-manager) if in App scope. Fix: annotate argocd.argoproj.io/compare-options: IgnoreExtraneous or global resource.exclusions for Secret kind (but then Git Secret changes won't sync). Better: use SealedSecret — Argo manages SealedSecret, controller creates Secret, Argo ignores Secret. (2) Don't commit stringData — cluster stores data (base64), diff forever shows mismatch. Configure ignoreDifferences for /data and /stringData JSON pointers, or only commit data (base64). (3) Sync waves ( argocd.argoproj.io/sync-wave: "-1") ensure ConfigMap/Secret before Deployment; usually Kind ordering suffices.

Change Process Templates

Small change (non-sensitive value): Git edit overlay → PR shows kubectl diff / kustomize → merge → Argo auto-sync (name hash triggers rollout) → observe 10min → rollback via Git revert. Preconditions: configMapGenerator with hash suffix, Argo prune/selfHeal safe, rollback verified by ConfigMap content SHA256. Medium change (new key): ship app code with default first → add key with default value → observe → change value. Large change (Secret rotation, encryption, external secrets): backup + rollback plan + staging validation → maintenance window → canary (1 replica/env) → batched rollout → verify function/metrics/logs → document.

Production Checklist

Naming : <app>-config / <app>-secret — check via kubectl get cm -A Labels : app.kubernetes.io/name, owner — enforced by admission policy

Sensitive data : ConfigMap has no credentials — CI + admission

Consumption : Files via directory mount, no subPath — daily scan item 4

Missing tolerance : No optional: true on primary configs — daily scan item 5

Permissions : Private keys 0400 no fsGroup; others 0440 + fsGroup — check items[].mode / fsGroup Update mechanism : Name hash / Reloader / app hot reload — inspect Pod template

Rollback : Deployment revisionHistoryLimit ≥3, config backups — kubectl get deploy -A -o json | jq ... Encryption at rest : Secret etcd prefix k8s:enc: — etcdctl get RBAC : No wildcards, list secrets minimal — auth can-i --list Audit : Enabled, logs Secret get/list — check apiserver args

Count quota : ResourceQuota per namespace — kubectl get quota -A Daily scan : Automated report — CronJob or external revisionHistoryLimit often overlooked (default 10). If set to 0/1, kubectl rollout undo has no history. It controls old ReplicaSet retention, not ConfigMap history.

Summary

ConfigMap/Secret management balances three tensions: (1) Flexibility vs Predictability — K8s default is "partially flexible" (volumes update, env vars don't, subPath never, latency varies). Choose one explicit mechanism: Kustomize hash + rollout, or app hot reload. (2) Security vs Convenience — etcd plaintext is convenient; encryption/KMS/external secrets add complexity. Match threat model: etcd snapshot access → encryption mandatory; only Git leak fear → Sealed Secrets sufficient. Don't deploy unmaintainable KMS for "looks secure". (3) Centralization vs Autonomy — single ConfigMap easy but wide blast radius; many small objects enable canary but complex refs. Balance: split by change frequency and impact — high-frequency tunables (log level, timeouts) separate keys; low-frequency high-risk (DB address) with credentials; config files via volume mounts.

Fixes for opening scenarios: (1) Key rename causing mass crash → CI validates app-read keys vs ConfigMap-provided keys. (2) Passwords always plaintext → encryption at rest first, then external secrets (order matters: unencrypted etcd makes external secret references sensitive too). (3) Cert rotation no effect → remove subPath, add nginx -t + reload sidecar or use ingress-nginx auto-reload. Rules aren't complex; sustaining compliance is. Hence engineering: structure, CI gates, admission policies, daily scans. Mechanisms replace vigilance — that's what "daily management practice" means.