From Push to Pull: Argo CD Multi-Env Governance & Safe Long-Running Task Delivery

0. Reading Boundaries, Version Baseline & Terminology

The article targets Go/Java backend engineers, platform engineers, SREs, and Kubernetes operators. Its core thesis: a successful pipeline run ( helm upgrade) or Argo CD Synced status alone does not prove safe delivery of long-running task systems. In Aegis-like agent platforms, new Pods may be Ready while old Workers still call external tools; databases may have migrated while old versions still renew leases and write checkpoints; rolling back images may leave old programs unable to read new task states. If external tool calls succeeded but results not yet persisted, re-execution can cause duplicate payments, notifications, or writes.

Therefore release governance is not merely replacing deploy commands with GitOps, but closing the causal chain in design, implementation, and drills:

Approved immutable release manifests

Constrained cluster convergence via Argo CD

Recoverable Worker execution with lease/fencing

Controlled data evolution with checkpoints, idempotency, and reconciliation

Auditable release conclusions

The article emphasizes Argo CD handles declarative resource comparison and convergence; runtime protocols, database migrations, and side-effect recovery remain joint responsibilities of application and platform.

0.1 Component Version Matrix

Do not use floating stable documentation as acceptance criteria. Every platform upgrade or process change must register actual versions and dependent capabilities in a release baseline. The matrix includes Kubernetes (Deployment, HPA, PDB, pod termination semantics, admission), Argo CD (Application, AppProject, Hook, Sync Window, RBAC), Helm (values merging, schema, template rendering), PostgreSQL (conditional updates, row locks, concurrent indexes, transactions), Admission Controllers (image signature/proof verification, exemptions, audit), and Worker Runtime (lease, epoch, checkpoint, idempotency). Each row should have an owner, verification date, and drill record link.

0.2 Key Terminology

Release Identity : A collection of source code, image, config, database, and protocol versions linked by a release_id.

Approved SHA : The reviewed and pinned config repo commit SHA; not the commit main happened to point at.

Convergence : Argo CD making cluster resources match Git-declared state; not equal to business task safety.

Drain : Stop accepting new tasks while completing, checkpointing, or safely abandoning in-flight tasks within a budget.

Lease : A time-bound grant of task execution eligibility to a Worker.

Fencing / Epoch : An incrementing execution generation on each takeover; old generations cannot submit state.

Contract : A breaking database or protocol change (e.g., dropping a column, tightening constraints).

1. Define Release Success Before Choosing Tools

Aegis API receives tasks; Workers pull from PostgreSQL or message systems, call models and external tools, and persist execution state. Tasks last seconds to hours; Dev, Staging, Production have different capacities, permissions, and data boundaries. Early push-based CI patterns mixed build, deploy, and cluster admin identities, lacking environment diff review, runtime fix write-back, and cross-version recovery protocols.

Adopting pull delegates continuous convergence to an independent controller, removing write access to target clusters from build pipelines and establishing an auditable entry point for desired state. It is not a magic substitute for runtime correctness.

1.1 Three-Layer Success Conditions

Declarative Convergence : Target resources match approved version. Evidence: Application revision, operation, diff. Failure means cluster didn't deploy approved desired state.

Workload Availability : New Pods schedulable, started, and capable of accepting tasks. Evidence: Deployment conditions, probes, dependency connectivity. Failure means new executors not yet available.

Business Recovery Correctness : In-flight tasks complete or are taken over; side-effects not duplicated. Evidence: Lease, epoch, checkpoint, idempotency, reconciliation records. Failure risks omissions, duplicates, or unknown results. Synced only describes desired vs actual resource comparison; Healthy depends on resource health checks. Neither alone proves "tasks not lost" or "external side-effects not repeated".

1.2 Four Pre-Release Invariants

Only a Worker holding a valid lease with matching epoch may submit task state.

Checkpoint, state changes, and execution eligibility updates must be atomically written in the same transaction or conditional write to establish causality.

External side-effects must have stable operation IDs, idempotent handling, or queryable results; otherwise they enter manual reconciliation queues.

Any rollback-allowed version must read current DB schema, task protocol, and checkpoints; if not, fix forward or route by protocol instead of blindly rolling back images.

Extending terminationGracePeriodSeconds only reduces interruption probability; it cannot replace these invariants.

2. Architecture Boundaries: Git Manages Desired State, Runtime Manages Execution Facts

The flow: Business Source → CI (test, build, sign, attest) → Image Registry (immutable digest) → Config PR (validation & approval) → Release Manifest (approved SHA + release_id) → Config Repo → Argo CD (compare & sync) → Kubernetes (RBAC & admission) → Worker (acquire, drain, recover) → PostgreSQL (tasks, leases, checkpoints) → External Tools (idempotent, queryable, reconciliation).

Ownership separation:

Deployment templates, resource boundaries → Argo CD manages; Git stores Helm/Kustomize declarations; not live Pod state.

HPA current replicas → HPA manages; Git stores HPA spec; not live Deployment replicas.

Secret contents → Secrets system & sync controller; Git stores encrypted ciphertext or external refs; not plaintext credentials.

Tasks, Leases, Checkpoints → Runtime & DB; Git stores schema & protocol versions; not runtime records.

Data fixes → Audited ops process; Git stores fix programs, change tickets, execution evidence; not unaudited historical data replay.

Cluster security policies → Platform apps; Git stores independent policy & audit config; not business apps relaxing policies.

Git can store reliably governed ciphertext; "Git only stores Secret refs" is not the definition of GitOps. What matters: who can decrypt, who can sync, how to verify actual access remains normal after rotation.

3. Release Objects Must Be Immutable: From "Main Branch State" to "Approved SHA"

3.1 Release Identity Contains Multiple Versions

source_commit

: traces business source code. image_digest: pins image content. config_commit: pins Chart, values, Application, policies. schema_version: pins DB migration state. task_protocol_version: pins task & checkpoint encoding compatibility. release_id: links approval, sync, metrics, task events, alerts.

Record as immutable ReleaseRecord (example YAML shown). Not required to self-reference config_commit in same Git commit; release system can generate record after merge, writing approval object, merge commit, Argo CD operation revision, actual Pod digest into tamper-proof audit system.

3.2 Production Must Not Track Floating main After Approval

If production Application targetRevision is main, any new commit between approval and manual sync could deploy unapproved content. Specifying revision once via CLI also separates declared target from actual sync result. Production must explicitly choose one pattern and codify it:

Pinned Commit : Application targetRevision updated via protected PR to full SHA. Clearest approval object; each release changes Application.

Protected Release Branch : Only release service updates releases/production. Intuitive flow; branch protection & rollback specs must be strict.

Signed Tag : targetRevision points to immutable verified tag. Human-readable; must prevent tag rewrite and verify signatures.

Article recommends pinned commit. Production release manifest updates targetRevision with PR reviewing digest, protocol scope, schema, and acceptance evidence. Do not treat main as approval object.

3.3 Two Mandatory Production Control Points

Allow becoming production target version : Production directory, release record, Application revision PR approval, branch protection, CODEOWNERS.

Allow starting production sync : Argo CD RBAC, Sync Window, release on-call process, single-app mutex.

"Merge-to-deploy" and "merge-then-manual-sync" are both viable; but cannot mix approval model, Argo policies, and emergency processes. Article adopts: non-prod auto-sync, production pinned SHA then manual sync.

4. Multi-Environment Config & ApplicationSet: Environment Mapping Must Be Auditable

Recommend separating business source from deploy config, but repo count should follow team size. Core invariant: production config write permissions, approval, audit must be independent. Promote same verified image digest, not "same source rebuilt per environment". Still must review env config, Chart, DB, protocol diffs; copying digest ≠ copying full runtime environment.

4.1 Non-Production ApplicationSet

Avoid unconstrained Matrix creating Cartesian product of "all environments" × "all workload clusters". With fixed three-env mapping, List generator is easier to audit. Example ApplicationSet uses list generator for dev/staging targeting same staging-cluster, with goTemplate, valueFiles per env, automated sync with prune/selfHeal, ignoring HPA replica differences. Assumes Chart creates Deployment named aegis-worker, namespaces pre-created by platform. List order doesn't imply sequential promotion; env promotion controlled by PR or release system. If non-prod uses prune, include audit, confirmation, recovery strategy for deletions in drills; don't treat as risk-free convenience.

4.2 Production Application: Pinned Approved Revision

Production Application has no syncPolicy.automated, so no auto-sync. Before operation, release system must verify: Application revision, ReleaseRecord configCommit, operator-selected Application, current operation empty, no other in-progress release for same app. After sync, verify Argo CD operation revision, Deployment template digest, running Pod digest, release annotations.

5. Authorization Boundaries: AppProject Is Not the Entire Permission Model

AppProject restricts Application repos, destinations, resource kinds; cannot replace Git protection, Argo CD user RBAC, target cluster Kubernetes RBAC, workload identity governance. Example minimal AppProject for production: sourceRepos limited to config repo, destinations to prod cluster/namespace, namespaceResourceWhitelist to Deployment, Service, ConfigMap, Job, HPA, PDB. Role

production-release</strong> allows get/sync only on <code>aegis-worker-production

, bound to OIDC group oidc:aegis-production-release. Actual Argo CD RBAC object format, SSO group claims, policy scope must be validated in version baseline.

5.1 Sync Window: Time & Change Risk Control Plane

Production should define allowed windows, with audited exception paths for emergencies. E.g., routine changes only during staffed hours; DB Contract, permission changes, admission policy changes need stricter windows. Window is not approval substitute; reduces "unmonitored release" risk. Example syncWindows allow weekdays 20:00-22:00 Asia/Shanghai, 2h duration, manualSync true, syncOverrun true (allows started sync to continue past window end). Priority of allow/deny, timezone, manual sync behavior, emergency exemptions must be validated per actual Argo CD version. Production process still needs external change tickets & audit to avoid "window allows" being misread as "anyone can deploy".

5.2 Additional Boundaries to Review

Config repo protects production dir, project, target cluster registration, root app maintained by platform.

Argo CD target cluster credentials follow least Kubernetes RBAC; avoid default cluster-admin.

Worker uses dedicated ServiceAccount; set automountServiceAccountToken: false when no K8s API access needed.

Admission policies restrict privileged containers, hostPath, dangerous capabilities, unapproved images.

Same namespace must not allow arbitrary Pods to bind high-privilege ServiceAccounts; resource whitelist cannot prevent identity escalation.

6. Helm, HPA & Declarative Ownership

Argo CD uses Helm to render manifests then manages K8s objects; Helm does not manage release lifecycle in this chain. Do not mix helm upgrade, helm rollback, and Argo CD on same resource. Values follow normal Helm merge semantics; multiple values files not a bypass for merge rules—scrutinize list overrides, nulls, defaults, schema-constrained final render.

6.1 replicas Must Have Single Owner

Template conditionally omits replicas when autoscaling.enabled true. With HPA enabled, HPA manages replicas. Historical manifests declaring replicas may trigger scale-down on first handoff; must validate in non-prod, not rely on ignoreDifferences to mask risk. ignoreDifferences excludes field from diff; RespectIgnoreDifferences=true makes sync respect that rule. Primarily resolves existing resource field ownership conflicts; not "auto-delete field on first create". Ignore rules must target specific Deployment to avoid masking manual scale-down of ordinary services.

6.2 Rendering & Diff Checks Are Release Inputs, Not Conclusions

Commands: helm lint, helm template, argocd app diff, argocd app get --refresh. Additional checks: target cluster/namespace, pinned digest, resource whitelist, deletion scope, Secret refs, HPA/PDB coordination, requests/limits, drain budget, NetworkPolicy. helm lint and server-side dry-run cannot prove real rolling update, task takeover, external side-effect recovery correctness.

7. Supply Chain: Digest Determines Content, Signatures & Attestations Determine Provenance

Rebuilding per environment makes Staging-verified artifact differ from Production-running artifact. Correct path: build once, sign & verify once, promote with same digest. Values use image.digest (required), values.schema.json validates digest format, CI rejects tag-only configs. Build side reads digest from actual push output; don't rely on local dev tags or unstable RepoDigests.

7.1 Admission Policy Acceptance Contract

No untested generic Kyverno/Cosign policies given, but production teams must define: image scope (allowed registry/repo/digest forms), signature identity (allowed issuer, identity, key rotation), provenance (source repo, build workflow, branch/commit constraints), vulnerability exceptions (risk level, approver, expiry, compensating controls), policy failure behavior (fail-open/closed), multi-arch images (sign index or platform manifest, how admission parses). Evidence: rejected/allowed test results, signature verification logs, provenance/SBOM refs, exception tickets & expiry tests, failure drill results, per-platform pull verification. Argo CD does not auto-verify signatures just because digest present; verifying only internal registry while allowing arbitrary external images breaks trusted supply chain.

8. Database Releases: Expand First, Contract Separately Approved

Old version write failures often stem from changes breaking new/old coexistence protocol, not migration "not late enough". Four phases:

Expand : Add compatible columns/tables/indexes; old versions continue read/write; rollback condition: old code still reads/writes.

Migrate : Batched backfill & verification; new/old paths compatible; rollback: retain old fields & conversion paths.

Switch : Switch read/write paths; new path becomes primary; rollback: old version still understands new data.

Contract : Tighten constraints, drop old structures; no longer depend on old protocol; rollback: explicitly close rollback window.

"New column with default" not guaranteed safe; evaluate locks, default expressions, table size, PostgreSQL version. DB backward compatibility ≠ Checkpoint, tool params, task encoding backward compatibility.

8.1 PreSync Only for Short, Re-entrant Expand

Example PreSync Job runs migrator image with expand arg, backoffLimit: 0, activeDeadlineSeconds: 300, dedicated ServiceAccount with minimal DDL perms. Prerequisite: migrator implements DB mutex, migration version table, script checksum, failure exit, traceable logs. IF NOT EXISTS only checks name, not structure correctness; re-execution must verify final structure & checksum. Static Job name + BeforeHookCreation requires serialized release process. If previous Job not finished, new release must not delete/replace its basis. Implement per-Application mutex at release controller layer; define evidence retention & cleanup for Job success/failure/timeout.

8.2 PostSync & Sync Wave Boundaries

PostSync

runs after successful sync & resource health; does not prove old Pods exited, old leases/partitioned Workers/external calls invalidated. Selective sync skips hooks; releases depending on migrations must not bypass full sync. Contract must be independent change, checking: old Pods & ReplicaSets, Worker heartbeats, active leases, offline tasks, CronJobs, rollback-allowed versions, other shared DB clients. observedGeneration only means controller observed desired state; sync wave orders resource apply, cannot provide cross-process write mutex.

8.3 Large Indexes & Long Backfills Run Independently

CREATE INDEX CONCURRENTLY

cannot be in regular transaction block; failure may leave invalid index; name existence ≠ index valid. Hour-long backfills must not block app sync; delegate to migration process with progress reporting, throttling, pause, retry, reconciliation. DB downgrade prefers forward fix; PITR is full-cluster disaster recovery, not image rollback button.

9. Long-Task Rolling Releases: Stop Acquiring, Then Bounded Drain

9.1 Readiness Cannot Replace Acquisition Gate

Readiness affects Service traffic, does not stop Worker pulling tasks from PostgreSQL or queue. Pure pull-based Workers must execute atomic drain gate in acquisition loop. State machine: Accepting → (SIGTERM or admin) → Draining (in-flight tasks complete, reach drain deadline, cooperative exit & safe commit, or cannot confirm safe handoff → exit, takeover by new Worker) → Finished → Checkpointing → LeaseExpiry. Drain start refuses new acquisitions; executed tasks keep independent execution context, renew lease while eligible. At internal deadline, cancel execution, wait cooperative exit; only tasks confirmed locally stopped can be actively handed over. Cannot set task back to PENDING expecting old goroutine to stop, because new Worker may have already taken over.

9.2 Go Context Layering

Example runWorker uses signalCtx for shutdown trigger only; separate claimCtx (acquisition loop) and execCtx (task execution). Start registers in-flight before execution; registration & drain gate under same lock. On signal: BeginDrain() atomically closes acquisition gate; /readyz returns cannot-accept-tasks. stopClaim(), wait claim loop stopped (I/O with timeout). Wait in-flight 7 min; if done return nil. Else stopExec() (cooperative cancel tool calls; remote side-effects not auto-revoked). Wait in-flight 30 sec; if not done return ErrDrainDeadline (no bulk reset to PENDING; exit, lease expiry & fencing handle takeover). Implementation must ensure: tasks acquired but not yet executing count as in-flight; no concurrency window between drain start & new task registration; DB & tool HTTP have timeouts; after context cancel, recoverer queries/retry remote side-effects by idempotency key.

9.3 Kubernetes Termination Budget

Deployment strategy: RollingUpdate, maxSurge 1, maxUnavailable 0, minReadySeconds 10, progressDeadlineSeconds 1200, terminationGracePeriodSeconds 600. Probes: readiness /readyz (5s), liveness /livez (10s), startup /livez (5s, failureThreshold 30). Not complete Deployment; need selector, labels, image, resources, securityContext, NetworkPolicy. Liveness should not fail due to normal drain or brief DB blip. Simple preStop: sleep 30 may let pull Workers continue acquiring; usually less clear than proper SIGTERM handling. Budget example: stop acquire 10s, normal drain 420s, cancel & save 30s, exit margin 140s = 600s. Config sample, not performance promise; hour-long tasks must rely on recoverable checkpoints, not infinite grace period. maxUnavailable: 0 constrains Deployment rolling update availability drop, not business zero-downtime promise. PDB constrains voluntary evictions via Eviction API, not directly Deployment rolling updates, cannot prevent node failure or direct Pod deletion.

10. Lease, Fencing & Side-Effects: Last Mile of Correctness

Checkpoint decides recovery point, Lease decides temporary executor, Fencing decides if expired executor can still write. Three are not interchangeable.

10.1 Acquisition & Takeover Must Be Atomic

Task table stores owner_id, epoch, lease_until, status, checkpoint, protocol_version. Conceptual PostgreSQL acquisition: CTE selects candidate (PENDING/RETRY or RUNNING with expired lease) ordered by priority, FOR UPDATE SKIP LOCKED LIMIT 1; then UPDATE sets owner, increments epoch, sets lease_until, status=RUNNING, updated_at, returning epoch, lease, checkpoint, protocol_version. Key: candidate selection, row lock, owner set, epoch increment, lease set, return execution eligibility all in single atomic operation. Production impl must define priority fairness, empty-acquisition backoff, transaction timeout, deadlock retry, partitioning/index strategy.

10.2 Renewal, Progress, Result Submission All Carry Eligibility Conditions

Update checkpoint, lease_until, updated_at WHERE id=$3 AND owner_id=$4 AND epoch=$5 AND status='RUNNING' AND lease_until > now(). Zero rows affected means eligibility lost; Worker must stop further state submissions & new business steps, not retry same ineligible write. Final success, failure, voluntary handoff use same owner/epoch/lease conditions. Voluntary handoff only after confirming local execution stopped; submit failure cannot claim handoff success.

10.3 DB Fencing Does Not Auto-Constrain External Tools

Failure points & recovery actions:

Exit before tool call → resume from last committed step.

Tool succeeded & result persisted → read result, skip re-execution.

Tool succeeded but result not persisted → query by stable operation ID or idempotent retry.

Tool lacks query & idempotency → mark result unknown, enter manual/compensation reconciliation.

Idempotency key should represent business op, e.g., task_id + step_id + operation_version. Cannot use new epoch as idempotency key, else each takeover looks like new op to downstream. Controlled downstreams may verify fencing token; third-party services rely on idempotency keys, Outbox, reconciliation. Outbox only guarantees local fact & delivery intent consistency, cannot alone promise exactly-once for arbitrary third parties.

10.4 Rollback Must Also Check Protocol Scope

If new Worker wrote checkpoints old version cannot read, rolling back image may make tasks unrecoverable. Release manifest should declare read/write protocol range; options: versioned envelope, compatible reads, protocol-based Worker routing, or complete new-protocol tasks before rollback. No destructive DDL in DB ≠ safe image rollback.

11. Quantitative Health Gates & Release Decisions

Metrics without thresholds, observation windows, action ownership are not executable gates. Example gates (thresholds must be replaced with system baselines):

Infrastructure (10 min): available replicas, image pull, probes, resource pressure. Block if new version no available replicas; CrashLoop; scheduling fails 5 min. Action: stop subsequent batches; platform/SRE investigate.

Lease Runtime (cover ≥1 lease cycle): renewal failure rate, takeover P99, old epoch write rejections. Block if renewal failure exceeds baseline; takeover P99 exceeds SLO; old epoch write succeeds. Action: pause acquisition or promotion; app owner handles.

Business Semantics (cover representative long tasks): unknown results, duplicate side-effects, protocol errors. Block if any confirmed duplicate side-effect; unknown result no reconciliation path; protocol parse errors >0. Action: stop new task assignment; incident classification & fix.

Database (during & after migration): lock waits, error rates, checksum. Block if schema version/checksum mismatch; lock wait exceeds budget. Action: stop sync; DBA & service owner joint handling.

Supply Chain (pre-deploy): signature/attestation/scan/admission events. Block if verification fails; exception expired; policy bypass. Action: reject release; platform security handles.

Put task_id in logs/traces, not Prometheus labels (high cardinality). Metrics aggregated by env, release_id, Worker version, task type. Long tasks have feedback lag; observation window must cover representative tasks, not just recently completed short tasks. For pull Workers, canary by queue/tenant/task type/acquisition condition; HTTP traffic weight won't auto-distribute PostgreSQL tasks proportionally. Argo Rollouts helps orchestrate but cannot replace task assignment protocol.

12. Complete Production Release Execution Checklist

CI tests source, builds image once, records digest, source commit, SBOM, signature, provenance.

Submit Dev config PR; run schema, render, resource boundary, deletion scope checks.

Dev auto-sync; verify acquire, renew, idempotency, drain, recovery paths.

Promote same digest to Staging; run mixed-version, failed migration, node drain, old Worker revive, capacity drills.

Create production release record: pin digest, schema, protocol read/write range, approved config_commit, evidence links.

Production PR only points Application to approved immutable revision; review diff, forbid ad-hoc param overrides.

Validate RBAC, Sync Window, per-app mutex, current operation, admission policy status.

Execute full sync; forbid selective sync bypassing migration hooks.

Verify operation revision, actual Pod digest, migration records, infra gates, runtime gates, business gates.

Retain rollback-compatible image & DB schema until observation window ends; Contract requires separate approval.

CI's main duty: write to protected Git. If process calls Argo CD, release identity must be separate from build identity, scoped to specific production app & allowed actions.

13. Rollback & Emergency: Confirm Rollback Object First

Rollback strategies per object:

Image, ordinary config : Update to previous approved immutable revision & sync. Constraint: verify schema & protocol compatibility.

Executed Expand : Usually retain compatible structures. Constraint: avoid unnecessary reverse DDL.

Contract, bad data : Dedicated fix plan, restore if needed. Constraint: not auto-reverted by manifest.

External side-effects occurred : Idempotent query, reconciliation, compensation. Constraint: image swap cannot undo payments/messages.

Prefer Git fix: revert only this promotion's changes, avoid rolling back other apps or security fixes. With production auto-sync off, commit revert then controlled sync. If Git path unavailable, break-glass with scoped app, owner, TTL, recovery steps: 1) Identify if need to stop new task acquisition, not just pause resource sync. 2) Pause other release entrypoints for same app; ApplicationSet-managed apps handled at upper policy. 3) Record current revision, resource state, image, active leases, task impact scope. 4) Make minimal fix, save commands, operator, evidence. 5) ASAP write final state back to Git, review diff, restore convergence. Pausing auto-sync doesn't stop started operations, DB Jobs, Workers, third-party calls; each needs own handling. Auto-sync apps cannot safely rollback directly; must handle auto-sync policy & Git desired state first.

14. Failure Drills: Validate Governance with Evidence, Not "Last Success"

All drills in isolated env, test tool endpoints, non-prod data. Key experiments:

Approved SHA : Commit unrelated change to main after approval → Production Application still points to approved SHA, no new deploy.

RBAC : As releaser, try modify Project or sync other app → Rejected; only target app get/sync allowed.

Env Mapping : Generate ApplicationSet children → Only Dev/Staging created, correct cluster/namespace.

Production Policy : Modify production target without manual sync → App shows OutOfSync, workloads don't self-upgrade.

HPA Field : Scale then full sync → replicas not unexpectedly reset.

Image Admission : Deploy unapproved test image → Admission rejects; old available workloads not prematurely evicted.

Expand Failure : Migrator returns non-zero → Sync doesn't update subsequent workloads; error evidence retained.

Migration Re-entry : Repeat full sync for same migration → Schema version & checksum consistent, no duplicate destructive execution.

Graceful Termination : Delete running test Pod → Stops new acquisition; in-flight complete or recover from checkpoint.

Non-Graceful Failure : Make Worker unable to submit then exit → Lease expiry triggers takeover, not relying on preStop.

Old Worker Revive : Pause old executor until after takeover, resume → Old epoch writes rejected, side-effects not duplicated.

Unknown Result : Simulate client timeout after tool success → Query or idempotent retry; if indeterminate, enter reconciliation.

Protocol Rollback : Start old version after new wrote checkpoint → Compatible read, or explicit reject & route compatible Worker.

PDB Boundaries : Execute node drain vs rolling update separately → Observe two mechanisms constrain differently.

Each drill retains: release SHA, image digest, task events, owner/epoch, migration records, admission events, metric screenshots, expected vs actual comparison. Observing eventual task success insufficient to prove no duplicate side-effects occurred mid-process.

15. Control Plane Recovery & Rollout Sequence

When Argo CD down, existing K8s workloads usually keep running; but new releases, drift repair, audit entrypoints affected. Must retain or rebuild: Application/AppProject, cluster registration identity, repo credentials, SSO/RBAC, admission policies, image retention rules, Git repo access. Three often-overlooked failure domains:

Erroneous prune : Dir deletion, generator changes, app deletions have different delete paths; production deletions need independent review.

Credential expiry : After rotation, verify control plane actually accesses target clusters & repos, not just Secret updated.

Controller contention : Same resource cannot be managed by multiple Applications, Helm CLI, or external scripts.

Suggested rollout phases with verifiable artifacts:

Convergence Entry : Config repo, digest, explicit Application → Rebuild workloads from Git & trace artifacts.

Approval Object : ReleaseRecord, pinned revision, protection rules → Post-approval main changes don't affect this production target.

Permission Boundaries : Project, RBAC, Sync Window, cluster creds, admission → Privilege escalation & unapproved artifacts rejected.

Safe Drain : Stop-acquire gate, context layering, probes, termination budget → No new tasks started during SIGTERM.

Recoverable Execution : Atomic acquire, lease, epoch, idempotency, reconciliation → SIGKILL & old Worker revive both handled.

Compatible Releases : Expand/Contract, protocol range, rollback checklist → New/old coexistence & protocol rollback experiments pass.

Quantitative Governance : Release gates, thresholds, dashboards, on-call ownership → Failures block promotion & trigger explicit actions.

Official References & Evidence Boundaries

Links for mechanism verification (not proof article examples passed real cluster validation). Switch to actual installed version docs & complete article drills. References: ApplicationSet List generator, Argo CD Projects, RBAC, Sync Windows, Helm, Sync Options, Sync Phases/Waves, Automated Sync Policy, Kubernetes Pod Lifecycle, Kubernetes Disruptions, PostgreSQL CREATE INDEX.

Ultimately, Aegis safe release depends on complete causal chain: Approved immutable Git version decides desired resources, trusted digest decides executable code, constrained permissions decide who can start sync, lease & fencing decide who can execute, checkpoint, idempotency & reconciliation decide how to recover. Only when these links close together does "new Pod Ready" further translate to "this release is safe for tasks and business".