Build a Zero‑Trust Container Security Pipeline in 10 Practical Steps

Applicable Scenarios & Prerequisites

Applicable Scenarios :

Production Kubernetes clusters (any size)

Environments requiring security compliance (e.g., GB/T 22239, ISO27001, SOC2)

Multi‑tenant container platforms (SaaS/PaaS)

Supply‑chain security requirements (SBOM, image signing)

Prerequisites :

Kubernetes 1.24+ (supports Pod Security Standards)

Docker/Containerd runtime

Cluster‑admin privileges for installing security components

Private image registry (Harbor/Artifactory)

Basic monitoring stack (Prometheus/Grafana)

Environment & Version Matrix

Component

Version Requirement

Key Feature Dependencies

Minimum Resource Specs

Kubernetes

1.24+

PSS/PSA, RuntimeClass

-

Trivy

0.45+

SBOM generation, VEX support

1C2G (scan node)

Falco

0.36+

eBPF driver, rule hot‑load

512M/node

OPA Gatekeeper

3.13+

External data sources, mutation webhook

500M

Cosign

2.2+

Keyless signing, Rekor transparency log

-

Harbor

2.9+

Image proxy cache, signature verification

4C8G

Falco Sidekick

2.28+

Multi‑channel alert forwarding

256M

Quick Checklist

Step 1: Image supply‑chain security (signing + SBOM)

Step 2: Image vulnerability scanning & admission control

Step 3: Minimize base images (Distroless/Alpine)

Step 4: Configure Pod Security Standards

Step 5: Deploy OPA Gatekeeper policies

Step 6: Network policies & zero‑trust networking

Step 7: Secrets management & encryption

Step 8: Runtime threat detection (Falco)

Step 9: Audit logging & compliance

Step 10: Continuous security monitoring & response

Implementation Steps

Step 1: Image Supply‑Chain Security

Goal : Ensure image provenance and prevent supply‑chain attacks.

Install Cosign (image signing tool)

# RHEL/CentOS/Ubuntu generic
wget https://github.com/sigstore/cosign/releases/download/v2.2.0/cosign-linux-amd64
chmod +x cosign-linux-amd64
sudo mv cosign-linux-amd64 /usr/local/bin/cosign

# Verify installation
cosign version
# Output: cosign version v2.2.0

Generate signing keys

# Method 1: Local key pair (traditional)
cosign generate-key-pair
# Generates cosign.key (private) and cosign.pub (public)

# Method 2: Keyless signing (recommended, no key management)
export COSIGN_EXPERIMENTAL=1
cosign sign --oidc-issuer=https://oauth2.sigstore.dev/auth myregistry.com/myapp:v1.0.0

Sign the image

# Sign with local key
cosign sign --key cosign.key myregistry.com/myapp:v1.0.0
# Verify signature
cosign verify --key cosign.pub myregistry.com/myapp:v1.0.0

Generate SBOM (Software Bill of Materials)

# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
# Produce SPDX SBOM
syft myregistry.com/myapp:v1.0.0 -o spdx-json > sbom.spdx.json
# Attach SBOM to image
cosign attach sbom --sbom sbom.spdx.json myregistry.com/myapp:v1.0.0
# Sign SBOM
cosign sign --key cosign.key --attachment sbom myregistry.com/myapp:v1.0.0
# Verify SBOM
cosign verify-attestation --key cosign.pub --type spdx myregistry.com/myapp:v1.0.0

Step 2: Image Vulnerability Scanning & Admission

Goal : Block images with high‑severity vulnerabilities from reaching the cluster.

Install Trivy

# RHEL/CentOS
cat <<EOF > /etc/yum.repos.d/trivy.repo
[trivy]
name=Trivy repository
baseurl=https://aquasecurity.github.io/trivy-repo/rpm/releases/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://aquasecurity.github.io/trivy-repo/rpm/public.key
EOF
yum install -y trivy

# Ubuntu/Debian
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/trivy.list
apt update && apt install -y trivy

# Verify
trivy --version

Scan images

# Scan local image
trivy image myregistry.com/myapp:v1.0.0
# Show only HIGH/CRITICAL
trivy image --severity HIGH,CRITICAL myregistry.com/myapp:v1.0.0
# Export JSON report
trivy image -f json -o report.json myregistry.com/myapp:v1.0.0

Integrate into CI/CD

# .github/workflows/security-scan.yml
name: Container Security Scan
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Run Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'table'
          exit-code: '1'
          severity: 'HIGH,CRITICAL'
      - name: Upload results
        uses: github/codeql-action/upload-sarif@v2
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'

Kubernetes admission control (Trivy Operator)

# Install Trivy Operator
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/v0.16.0/deploy/static/trivy-operator.yaml
# Create admission policy to reject CRITICAL images
kubectl create namespace trivy-system
kubectl apply -f - <<EOF
apiVersion: aquasecurity.github.io/v1alpha1
kind: ClusterVulnerabilityReport
metadata:
  name: block-critical-vulns
spec:
  severityLevel: CRITICAL
  action: Block
EOF

Step 3: Minimize Base Images

Goal : Reduce attack surface and vulnerability count.

Distroless (recommended)

# Multi‑stage build example
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-w -s" -o myapp .

FROM gcr.io/distroless/static-debian11:nonroot
COPY --from=builder /app/myapp /app/myapp
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app/myapp"]

Distroless advantages: only application and runtime dependencies, < 5 CVEs, no shell or package manager.

Alpine (lightweight)

# Alpine base image
FROM alpine:3.18
RUN apk add --no-cache ca-certificates tzdata && rm -rf /var/cache/apk/*
COPY myapp /usr/local/bin/myapp
RUN addgroup -g 1000 appuser && adduser -D -u 1000 -G appuser appuser && chown -R appuser:appuser /usr/local/bin/myapp
USER appuser
EXPOSE 8080
CMD ["/usr/local/bin/myapp"]

Image size comparison:

# ubuntu:22.04 base image: 450MB
# alpine:3.18 base image: 85MB
# distroless/static: 25MB ← recommended

Step 4: Pod Security Standards

Goal : Enforce pod‑level security best practices.

Enable Pod Security Admission

# Verify Kubernetes 1.25+ enables it by default
kubectl get pod -n kube-system kube-apiserver-* -o yaml | grep PodSecurity
# Expected output includes --enable-admission-plugins=...,PodSecurity

Configure namespace security level

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

kubectl apply -f production-namespace.yaml
# Test privileged pod (should be denied)
kubectl run privileged-pod --image=nginx --privileged -n production
# Error: pods "privileged-pod" is forbidden: violates PodSecurity "restricted:latest"

PSS level comparison

Level

Features

Use Cases

Privileged

No restrictions (allows privileged containers, hostPath, hostNetwork, etc.)

System components

Baseline

Blocks obvious privilege escalation but allows read‑only hostPath

Development / testing

Restricted

Enforces non‑root, read‑only root FS, no privilege escalation

Production (recommended)

Pod‑level security context

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:v1.0.0
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}

Step 5: OPA Gatekeeper Policy Engine

Goal : Enforce custom security policies such as image whitelists and resource limits.

Install Gatekeeper

# Install latest stable version
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.13.0/deploy/gatekeeper.yaml
# Verify installation
kubectl get pods -n gatekeeper-system
kubectl get crd | grep gatekeeper

Create constraint template (disallow "latest" tag)

# constraint-template-image-tag.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8simagetag
spec:
  crd:
    spec:
      names:
        kind: K8sImageTag
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8simagetag
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not contains(container.image, ":")
        msg := sprintf("Container %v uses 'latest' tag (implicit)", [container.name])
      }
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        endswith(container.image, ":latest")
        msg := sprintf("Container %v uses 'latest' tag (explicit)", [container.name])
      }

# Apply template
kubectl apply -f constraint-template-image-tag.yaml

Create constraint instance

# constraint-no-latest-tag.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sImageTag
metadata:
  name: no-latest-tag
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces:
    - "production"
    - "staging"

# Test policy
kubectl run test --image=nginx:latest -n production
# Expected error: admission webhook "validation.gatekeeper.sh" denied the request: [no-latest-tag] Container test uses 'latest' tag (explicit)

Image whitelist policy

# constraint-template-allowed-repos.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
  validation:
    openAPIV3Schema:
      properties:
        repos:
          type: array
          items:
            type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8sallowedrepos
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not startswith(container.image, input.parameters.repos[_])
        msg := sprintf("Container %v uses disallowed image repo: %v", [container.name, container.image])
      }

# constraint-no‑allowed‑repos.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: prod-allowed-repos
spec:
  match:
    namespaces: ["production"]
  parameters:
    repos:
    - "myregistry.com/"
    - "gcr.io/distroless/"

Step 6: Network Policies & Zero‑Trust

Goal : Implement least‑privilege network access control.

Default deny policy

# network-policy-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

kubectl apply -f network-policy-default-deny.yaml

Application‑level fine‑grained policy

# network-policy-frontend.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 9000
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

Step 7: Secrets Management & Encryption

Goal : Protect sensitive data and prevent leakage.

Enable static Secret encryption

# Generate encryption key
head -c 32 /dev/urandom | base64
# Create EncryptionConfiguration (replace placeholder with actual key)
cat <<EOF > /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <BASE64_KEY>
  - identity: {}
EOF
# Update kube-apiserver static pod manifest to include:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
# Restart apiserver (static pod will reload)
# Verify encryption
kubectl create secret generic test-secret --from-literal=password=secret123
ETCDCTL_API=3 etcdctl get /registry/secrets/default/test-secret | hexdump -C

Sealed Secrets

# Install controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
# Install kubeseal CLI
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-linux-amd64
chmod +x kubeseal-linux-amd64
sudo mv kubeseal-linux-amd64 /usr/local/bin/kubeseal
# Create sealed secret
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=Secure@2024 \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-db-creds.yaml
# Apply sealed secret
kubectl apply -f sealed-db-creds.yaml

External Secrets (Vault integration)

# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
# Configure Vault SecretStore
cat <<EOF > vault-secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp-role"
          serviceAccountRef:
            name: myapp-sa
EOF
kubectl apply -f vault-secretstore.yaml
# Create ExternalSecret
cat <<EOF > db-externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-creds
  data:
  - secretKey: username
    remoteRef:
      key: database/prod
      property: username
  - secretKey: password
    remoteRef:
      key: database/prod
      property: password
EOF
kubectl apply -f db-externalsecret.yaml

Step 8: Runtime Threat Detection (Falco)

Goal : Detect abnormal container behavior in real time.

Install Falco

# Helm install
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set tty=true \
  --set falco.grpc.enabled=true \
  --set falco.grpcOutput.enabled=true
# Verify
kubectl get pods -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50

Custom security rules

# custom-rules.yaml
- rule: UnauthorizedProcessInContainer
  desc: Detect processes not in allowed list
  condition: >
    spawned_process and container and not proc.name in (app, nginx, sh)
  output: Unauthorized process started in container (user=%user.name command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: WARNING

- rule: ContainerPrivilegeEscalation
  desc: Detect privilege escalation attempts
  condition: >
    spawned_process and container and proc.name in (sudo, su, setuid)
  output: Privilege escalation attempt detected (user=%user.name command=%proc.cmdline container=%container.name)
  priority: CRITICAL

- rule: SensitiveFileAccess
  desc: Detect access to sensitive files
  condition: >
    open_read and container and fd.name in (/etc/shadow, /etc/passwd, /root/.ssh/id_rsa)
  output: Sensitive file accessed (user=%user.name file=%fd.name container=%container.name)
  priority: ERROR

# Load custom rules
kubectl create configmap falco-custom-rules --from-file=custom-rules.yaml -n falco
kubectl edit deployment falco -n falco
# Add volumeMount:
#   - name: custom-rules
#     mountPath: /etc/falco/rules.d
kubectl rollout restart deployment falco -n falco

Alert forwarding with Falco Sidekick

# Install Sidekick
helm install falco-sidekick falcosecurity/falco-sidekick \
  --namespace falco \
  --set config.slack.webhookurl=https://hooks.slack.com/services/xxx \
  --set config.slack.minimumpriority=warning
# Configure Falco to send alerts to Sidekick (add env var FALCO_GRPC_HOSTNAME=falco-sidekick)

Step 9: Audit Logging & Compliance

Goal : Record all API operations to satisfy compliance requirements.

Enable Kubernetes audit logs

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
- level: Request
  verbs: ["create","update","patch","delete"]
  resources:
  - group: ""
    resources: ["pods"]
  - group: "apps"
    resources: ["deployments","statefulsets"]
- level: None
  verbs: ["get","list","watch"]
- level: Metadata

# Update kube-apiserver static pod manifest to include:
# --audit-policy-file=/etc/kubernetes/audit-policy.yaml
# --audit-log-path=/var/log/kubernetes/audit.log
# --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100
# View logs
tail -f /var/log/kubernetes/audit.log | jq '.'

Integrate Falco audit plugin

# Install Falco audit plugin
kubectl apply -f https://raw.githubusercontent.com/falcosecurity/plugins/main/plugins/k8saudit/deployment.yaml
# Add plugin config to Falco ConfigMap
# plugins:
#   - name: k8saudit
#     library_path: libk8saudit.so
#     init_config: ""

Step 10: Continuous Monitoring & Response

Prometheus + Grafana monitoring

# Install kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace
# Import Falco dashboard (Grafana ID: 11914) and Trivy Operator dashboard (ID: 16337)

Key security metrics:

# Critical vulnerable images
count(trivy_image_vulnerabilities{severity="CRITICAL"} > 0)
# Falco alert rate
rate(falco_events_total{priority="Critical"}[5m])
# Privileged pods count
count(kube_pod_container_status_running{security_context_privileged="true"})
# Pods not complying with PSS
count(kube_pod_labels{pod_security_kubernetes_io_enforce!="restricted"})

Automated response example (isolate offending pod)

# falco-response-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: isolate-pod
spec:
  template:
    spec:
      serviceAccountName: falco-response
      containers:
      - name: isolate
        image: bitnami/kubectl:latest
        command: ["/bin/sh","-c","POD_NAME=${ALERT_POD_NAME} NAMESPACE=${ALERT_NAMESPACE} && \
          kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-${POD_NAME}
  namespace: ${NAMESPACE}
spec:
  podSelector:
    matchLabels:
      pod-name: ${POD_NAME}
  policyTypes:
  - Ingress
  - Egress
EOF
          curl -X POST https://hooks.slack.com/services/xxx -d '{"text": "Pod ${POD_NAME} isolated due to security alert"}'"]
      restartPolicy: Never

Best Practices (10 Items)

Enforce image signature verification with Cosign and block unsigned images via admission webhook.

Prefer minimal base images (Distroless, Alpine) and avoid full OS images.

Integrate vulnerability scanning in CI/CD; block CRITICAL/HIGH findings and rescan weekly.

Require non‑root execution (runAsNonRoot: true) and set USER in Dockerfiles.

Apply default‑deny NetworkPolicy per namespace and whitelist necessary traffic.

Externalize secrets using Vault or Sealed Secrets; avoid plain Secrets.

Deploy Falco DaemonSet cluster‑wide; respond to alerts with priority ≥ WARNING.

Implement least‑privilege RBAC; remove cluster‑admin and audit roles regularly.

Retain audit logs for 90 days and forward to a SIEM (e.g., Splunk, ELK).

Conduct quarterly red‑team/blue‑team exercises simulating container escape, image poisoning, and supply‑chain attacks.