Building a Fully Automated GitOps Delivery Pipeline with Argo CD on Kubernetes

1. Determine delivery boundaries and permissions

CI handles testing, building, scanning and pushing images; the environment repository declares image tags and Kubernetes manifests. Argo CD pulls a specific revision, renders manifests, compares the actual state and synchronizes within scoped permissions. CI must not hold production admin rights and manual changes must not be applied to the same objects that Argo CD manages, otherwise OutOfSync masks real drift.

kubectl config get-contexts
kubectl config use-context <cluster-context>
kubectl version --client
kubectl cluster-info

If the API server is unreachable, verify kubeconfig, network, VPN and authentication, then record server version, nodes and current permissions for compatibility, scheduling or RBAC troubleshooting.

kubectl version -o yaml
kubectl get nodes -o wide
kubectl auth can-i create deployments -n <namespace>
kubectl auth can-i get pods -n <namespace>

Production should grant Argo CD write permission only to the target namespace; developers and CI trigger releases via protected branches or merge requests and retain read‑only troubleshooting rights.

2. Let the repository express all environment differences

Application templates and environment configurations can reside in the same repository or be split for permission reasons. In any case the image, resources, ingress and policies used in production must be reproducible from a single commit without relying on temporary local parameters.

platform-gitops/
├── apps/orders-api/
│   ├── Chart.yaml
│   ├── values.yaml
│   ├── templates/
│   └── values/
│       ├── test.yaml
│       ├── staging.yaml
│       └── production.yaml
└── argocd/
    ├── projects/
    └── applications/

Initialize the repo with protected branches, approvals and test gates. The production directory only accepts reviewed changes on main; CI must not force‑push past audits.

git init platform-gitops
cd platform-gitops
git checkout -b main
git config user.name "<author-name>"
git config user.email "<author-email>"
mkdir -p apps/orders-api/{templates,values} argocd/{projects,applications}

Chart version does not equal image version; the image tag must be explicitly managed in values. Production must forbid latest tags to avoid repository overwrite or node‑cache issues.

# apps/orders-api/Chart.yaml
apiVersion: v2
name: orders-api
description: GitOps managed API
type: application
version: 0.1.0
appVersion: "1.0.0"

# apps/orders-api/values.yaml
replicaCount: 2
image:
  repository: <image-repo>/orders-api
  tag: "1.0.0"
  pullPolicy: IfNotPresent
service:
  port: 8080
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

Production values contain only real environment differences. Requests drive scheduling; limits provide isolation and should be determined by capacity assessment and monitoring.

# apps/orders-api/values/production.yaml
replicaCount: 4
image:
  tag: "<immutable-image-tag>"
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

3. Write application manifests that can be correctly evaluated

Argo CD’s Synced status does not guarantee business availability. Deployments must include consistent labels, resource requests, readiness/liveness probes and a secure security context. If readiness fails, Pods never appear in Service endpoints, indicating a critical outage.

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "orders-api.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "orders-api.name" . }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "orders-api.name" . }}
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: orders-api
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - name: http
          containerPort: 8080
        resources: {{- toYaml .Values.resources | nindent 12 }}
        readinessProbe:
          httpGet:
            path: /readyz
            port: http
          initialDelaySeconds: 5
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
          initialDelaySeconds: 15
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true

A unified helper prevents mismatched Service selectors and Pod labels. When a Service has no Endpoints, compare selector and labels before deleting or restarting Pods.

{{/* templates/_helpers.tpl */}}
{{- define "orders-api.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end }}
{{- define "orders-api.fullname" -}}
{{- printf "%s-%s" .Release.Name (include "orders-api.name" .) | trunc 63 | trimSuffix "-" -}}
{{- end }}

# templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "orders-api.fullname" . }}
spec:
  selector:
    app.kubernetes.io/name: {{ include "orders-api.name" . }}
  ports:
  - name: http
    port: {{ .Values.service.port }}
    targetPort: http

ConfigMap changes that require process restarts must trigger a Pod template change; merely updating a ConfigMap does not restart containers that read environment variables.

# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "orders-api.fullname" . }}-config
data:
  LOG_LEVEL: "info"
  FEATURE_MODE: "stable"

# Add checksum annotation to force rollout on ConfigMap change
annotations:
  checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") | sha256sum }}

Before committing, run lint, render and client‑side dry‑run. Server‑side dry‑run validates admission webhooks without persisting resources.

cd apps/orders-api
helm lint .
helm template orders-api . --namespace <namespace> \
  -f values.yaml -f values/production.yaml > /tmp/orders.yaml
kubectl apply --dry-run=client -f /tmp/orders.yaml
kubectl apply --dry-run=server -n <namespace> -f /tmp/orders.yaml
kubectl diff -n <namespace> -f /tmp/orders.yaml

If a webhook rejects the manifest, save the resource, field and webhook name as evidence, then fix the Git configuration; never temporarily relax production policies to bypass checks.

4. Install and verify Argo CD

Installation creates CRDs, RBAC, controllers and services – a control‑plane change. Production must use a fixed, reviewed release tag, validate in an isolated environment, and check internal image registry, network policies, resource specs and change windows before applying.

kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/<version>/manifests/install.yaml
kubectl rollout status deployment/argocd-server -n argocd --timeout=10m

UI accessibility does not guarantee controller health; issues in repo-server, application-controller or Redis can block delivery.

kubectl get pods -n argocd -o wide
kubectl get deployments,statefulsets -n argocd
kubectl get events -n argocd --sort-by=.lastTimestamp
kubectl logs deployment/argocd-application-controller -n argocd --tail=100

The initial admin password is for first login only. Decoding the secret risks leakage; never copy it to tickets, logs or chats. Change it immediately after use or integrate SSO.

kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d
kubectl port-forward svc/argocd-server -n argocd 8080:443
argocd version --client
argocd login localhost:8080 --username admin --insecure
argocd account update-password
argocd account list

5. Use AppProject to limit blast radius

The default default project is overly permissive. Production Applications should restrict source repos, target clusters, business namespaces and allowed resource kinds. Even if an Application is altered, the project boundary prevents unauthorized targets.

# argocd/projects/orders-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: orders-production
  namespace: argocd
spec:
  sourceRepos: ["<repo-url>"]
  destinations:
  - namespace: <namespace>
    server: https://kubernetes.default.svc
  clusterResourceWhitelist: []
  namespaceResourceWhitelist:
  - group: ""
    kind: Service
  - group: ""
    kind: ConfigMap
  - group: apps
    kind: Deployment

AppProject resides in the argocd namespace, not the business namespace. When HPA, Ingress or ServiceAccount is required, add them after review; do not use wildcard whitelists.

kubectl apply --dry-run=client -f argocd/projects/orders-production.yaml
kubectl apply -n argocd -f argocd/projects/orders-production.yaml
kubectl get appprojects -n argocd
kubectl describe appproject orders-production -n argocd

Private Git repositories should use minimal‑permission read‑only deploy keys or robot tokens, never reuse personal PATs.

argocd repo add <repo-url> \
  --username <robot-account> \
  --password
argocd repo list

6. Application defines the full delivery contract

An Application binds the repo, revision, path, Helm values, project and target cluster. Protected main suits regular releases; strict freeze can lock a tag or commit SHA. For the first rollout, disable prune, observe the resource tree and assess deletion impact.

# argocd/applications/orders-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-api-production
  namespace: argocd
spec:
  project: orders-production
  source:
    repoURL: <repo-url>
    targetRevision: main
    path: apps/orders-api
    helm:
      valueFiles: [values.yaml, values/production.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: <namespace>
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
    syncOptions: [CreateNamespace=true]

selfHeal

corrects manual changes. Emergency edits should be committed back to Git or pause auto‑sync; otherwise the next reconcile will redeploy the Git version.

git status --short
git add apps/orders-api argocd
git commit -m "feat(gitops): add orders-api production application"
git push origin main
kubectl apply --dry-run=client -f argocd/applications/orders-production.yaml

Before the first sync, view the diff. argocd app diff returns a non‑zero status on differences; review the diff to ensure no unexpected resource deletions, selector changes or permission modifications.

kubectl apply -n argocd -f argocd/applications/orders-production.yaml
argocd app get orders-api-production
argocd app diff orders-api-production
argocd app resources orders-api-production

7. Kubernetes‑layer validation and troubleshooting

All kubectl commands must specify the namespace to avoid default‑namespace mis‑judgment. After Synced/Healthy, still verify rollout status, Pods, Endpoints, events and business endpoints.

kubectl rollout status deployment/orders-api-orders-api -n <namespace> --timeout=5m
kubectl get pods -n <namespace> -l app.kubernetes.io/name=orders-api
kubectl get svc,endpoints -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

If network and image policies allow, create a temporary curl Pod to verify intra‑cluster connectivity. The Pod is created and deleted; ensure the image comes from an approved internal registry and carries no production credentials.

kubectl run curl-check -n <namespace> --rm -i --restart=Never \
  --image=curlimages/curl:<image-version> \
  -- curl -fsS http://orders-api-orders-api:<port>/readyz

If an Endpoint list is empty, compare selector, labels and readiness probe – this often reveals the root cause better than a blind restart.

kubectl get service orders-api-orders-api -n <namespace> -o yaml
kubectl get pods -n <namespace> -l app.kubernetes.io/name=orders-api --show-labels
kubectl get endpoints orders-api-orders-api -n <namespace> -o yaml

When a resource stays Progressing for a long time, examine Deployment conditions, Pod events and previous container logs. Issues like ImagePullBackOff, FailedScheduling or CrashLoopBackOff require distinct handling.

kubectl describe deployment orders-api-orders-api -n <namespace>
kubectl get pods -n <namespace> -l app.kubernetes.io/name=orders-api
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c orders-api --previous --tail=200

8. CI updates only Git, not the cluster directly

After building an immutable image tag, CI should create a change or merge request in the environment repo; once approved, Argo CD pulls the revision. The script below syncs main, backs up values, renders Helm and shows the diff. The execution environment must lock tool versions.

#!/usr/bin/env bash
set -euo pipefail

REPO_DIR="<gitops-repo-dir>"
IMAGE_TAG="<immutable-image-tag>"
VALUES_FILE="$REPO_DIR/apps/orders-api/values/production.yaml"

cd "$REPO_DIR"
git fetch origin main
git checkout main
git pull --ff-only origin main
sed -i.bak -E "s|^(  tag: ).*|\1\"$IMAGE_TAG\"|" "$VALUES_FILE"
helm lint apps/orders-api
helm template orders-api apps/orders-api -f "$VALUES_FILE" > /tmp/orders-api.yaml
git diff -- "$VALUES_FILE"

Because sed in‑place editing differs across OSes, verify it on the executor first. Whether using sed or a YAML tool, limit field changes, validate rendering, and never force‑push past protected branches.

#!/usr/bin/env bash
set -euo pipefail

REPO_DIR="<gitops-repo-dir>>"
IMAGE_TAG="<immutable-image-tag>>"

cd "$REPO_DIR"

git add apps/orders-api/values/production.yaml
git commit -m "chore(release): orders-api $IMAGE_TAG"
git push origin HEAD:refs/heads/<release-branch>

If a push is rejected, it usually means the remote already has commits or protection rules are active. Re‑base onto the latest main or open a merge request; never force‑push to overwrite shared history.

9. Evidence‑based OutOfSync investigation

First confirm the revision, resource tree and events that Argo CD actually uses. If the revision differs from the expected Git SHA, check repository authentication, branch, cache and repo-server. If the revision matches but objects are abnormal, move to Kubernetes investigation.

argocd app get orders-api-production --refresh
kubectl get application orders-api-production -n argocd -o yaml
kubectl describe application orders-api-production -n argocd
argocd app resources orders-api-production

Controller logs need a time range and application name. Permission errors require checking AppProject, target‑cluster RBAC and controller identity; Git timeouts require DNS, NetworkPolicy, proxy or certificate verification.

kubectl logs deployment/argocd-repo-server -n argocd --since=30m | grep -F "orders-api-production"
kubectl logs statefulset/argocd-application-controller -n argocd --since=30m | grep -F "orders-api-production"

Manual edits, HPA replica changes or admission‑injected fields can cause OutOfSync. Locate the differing field first, then decide whether to fix Git, revert the manual change, or set a narrow ignoreDifferences rule. Do not hide unknown drift with a broad ignore.

# Use only when HPA controls replicas and the diff source is confirmed
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers: [/spec/replicas]

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa-name> -n <namespace>
kubectl get deployment orders-api-orders-api -n <namespace> \
  -o jsonpath='{.spec.replicas} {"desired,"} {.status.availableReplicas} {"available
"}'

Local reproduction must checkout the same SHA that Argo CD shows and use identical values. Uncommitted workspaces cannot serve as evidence.

git fetch origin main
git checkout <ArgoCD‑displayed‑SHA>
helm dependency build apps/orders-api
helm template orders-api apps/orders-api --namespace <namespace> \
  -f apps/orders-api/values.yaml \
  -f apps/orders-api/values/production.yaml > /tmp/revision-rendered.yaml

10. Automatic sync, deletion and rollback

When automatic sync is paused, modify the Git Application and then sync the change. UI‑only operations cause cluster state to drift from Git; re‑enabling auto‑sync may overwrite manual changes.

# Example without automated sync
spec:
  syncPolicy:
  syncOptions: [CreateNamespace=true]

argocd app get orders-api-production -o json | \
  jq '.spec.syncPolicy, .status.sync, .status.health'
kubectl get application orders-api-production -n argocd \
  -o jsonpath='{.spec.syncPolicy}
'

Before enabling prune, evaluate the deletion scope. If differences involve PVCs, shared Services, Ingress, Secrets or ConfigMaps, confirm ownership, backup or snapshot, and define a gray‑scale rollback plan.

argocd app diff orders-api-production
argocd app resources orders-api-production
kubectl get all -n <namespace> -l app.kubernetes.io/name=orders-api
kubectl get pvc -n <namespace>

Normal rollback prefers Git. Review the target commit to ensure it does not contain unrelated service, database migration or infrastructure changes.

git log --oneline -- apps/orders-api/values/production.yaml
git show --stat <commit‑SHA>
git show <commit‑SHA> -- apps/orders-api/values/production.yaml

#!/usr/bin/env bash
set -euo pipefail

REPO_DIR="<gitops-repo-dir>>"
COMMIT_TO_REVERT="<commit‑SHA>>"

cd "$REPO_DIR"
git checkout main
git pull --ff-only origin main
git revert --no-edit "$COMMIT_TO_REVERT"
helm lint apps/orders-api
helm template orders-api apps/orders-api \
  -f apps/orders-api/values/production.yaml >/tmp/orders-rollback.yaml
git push origin main

If Git is unavailable and an urgent stop‑loss is required, Argo CD history rollback can be used, but the Git repository must be reconciled immediately to avoid re‑deployment of the newer version.

argocd app history orders-api-production
argocd app rollback orders-api-production <history‑ID>
argocd app wait orders-api-production --sync --health --timeout 600

kubectl rollout status deployment/orders-api-orders-api -n <namespace> --timeout=5m
kubectl get pods -n <namespace> -l app.kubernetes.io/name=orders-api \
  -o jsonpath='{range .items[*]}{.metadata.name}\t{.spec.containers[0].image}
{end}'
kubectl get endpoints orders-api-orders-api -n <namespace>
argocd app get orders-api-production

11. Inspection and handover

Application status should be integrated into monitoring. Prometheus metric names and labels must match the actual exporter output; different Argo CD versions and ServiceMonitors may vary, so verify the metrics endpoint before solidifying alert rules.

# Use actual exporter metrics
argocd_app_info{sync_status!="Synced"}
argocd_app_info{health_status!="Healthy"}

Controller restarts and warning events are auxiliary evidence; still combine OOMKilled, logs, node pressure and Git connectivity to determine root cause.

kubectl get pods -n argocd \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount
kubectl get events -n argocd --field-selector type=Warning --sort-by=.lastTimestamp

During handover, preserve the revision, Argo CD history, Kubernetes rollout history, approval records and key monitoring links. The following commands collect read‑only evidence.

argocd app get orders-api-production
argocd app history orders-api-production
kubectl get deployment,service,pods -n <namespace> \
  -l app.kubernetes.io/name=orders-api
kubectl rollout history deployment/orders-api-orders-api -n <namespace>

Git describes what should be deployed; Argo CD revision and sync records show what the controller attempted; Kubernetes state, logs, metrics and business probes indicate actual availability. When all three evidence sources align, automated delivery is controllable; when they diverge, capture the differences and logs, narrow the impact and restore the desired state via Git.