AI Infra in Practice #11: Cross‑Cut Release Governance – Full Versioning of Prompts, Models, and RAG

1. Opening Incident

On a Friday, an operator edited a customer‑service Prompt, changing "please answer concisely" to "please give a conclusion with reasons". The change was deployed without a version tag. By Monday, average ticket handling time doubled, complaint rate rose 18%, and first‑contact resolution fell from 71% to 58%. A simultaneous RAG index rebuild added 5,000 documents, making it impossible to pinpoint whether the Prompt or the new index caused the degradation because neither had version identifiers.

This scenario is common: a 2026 Future AGI study reports that over 60% of LLM incidents stem from untested Prompt or model changes pushed directly to production. Traditional CI/CD only governs code, leaving Prompt, model, RAG index, tool schema, and workflow unmanaged—a major governance blind spot.

2. What Needs to Be Versioned

AI production agents involve six mutable objects:

Code : business logic, orchestration, inference gateway – managed by Git.

Prompt : system prompts, few‑shot examples, output constraints – changes occur ten times faster than code.

Model : base model version, fine‑tuned weights, embedding model – a single upgrade can shift recall by 5 pp or double toxicity.

RAG Index : document collection, chunking strategy, embedding version – rebuilding takes hours and rewrites all retrieval results.

Tool Schema : function‑calling parameter definitions – a field changing from optional to required can break the entire call chain.

Workflow : state‑machine, tool order, routing policy – a branch condition change rewires the inference path.

The governance system must answer three questions for each object: how to track versions, how to release safely (gray‑scale, statistical gate), and how to roll back instantly.

3. End‑to‑End Release Pipeline

The CI/CD boundary expands from "code only" to a unified pipeline that handles all six objects. The pipeline consists of seven gates:

Code CI : Git commit triggers unit tests, lint, and container build. The image label records SHA and the versions of Prompt, model, RAG index, tool schema, and workflow.

Prompt Version Check : Pull the current Prompt from a Prompt Registry (LangFuse or Vertex AI Prompt Registry) and run a Golden Set regression (500‑2000 samples). Accuracy, hallucination, and refusal rates must stay within ±2% of baseline; otherwise the change is blocked.

Model Change Verification : Detect a model alias change (e.g., champion v3 → v4) and launch a shadow comparison where both versions serve the same traffic. Metrics are collected in parallel.

RAG Index Validation : After a new index is built, evaluate Recall@5, MRR, and answer‑hit rate, and verify embedding dimension and field mapping compatibility.

L7 Gate (Evaluation Gate) : Aggregate results from Prompt, model, RAG, and schema checks. The gate returns PASS, BLOCK, or MANUAL REVIEW. A BLOCK stops the pipeline before any gray‑scale.

Gray Release : Ramp traffic 1% → 5% → 25% → 50% → 100%, holding each step for ≥24 h. Real‑time monitoring of quality, safety, latency, and business KPIs triggers automatic rollback if thresholds are crossed.

Full Release + Baseline Update : After successful gray, switch 100% traffic and write the new versions to the baseline store for the next cycle.

The core principle is "6 objects unified versioning + L7 gate + progressive gray + one‑click rollback".

4. Technical Deep‑Dive

4.1 Versioning Mechanisms

Code uses Git; the container image label stores all six version identifiers, enabling full‑artifact traceability.

Prompt is stored in a Prompt Registry (LangFuse, Vertex AI Prompt Registry, or Future AGI traceAI). Each Prompt has a version, label (production/staging/draft), and schema. Retrieval uses get_prompt(name, version). Lifecycle: Draft → Gated (eval‑on‑PR + canary) → Deprecation (archived + drain window).

Model lives in a Model Registry (MLflow). The alias system (champion/challenger/candidate) points to the active model. Rollback is an alias switch, instantaneous. Model signatures enforce input‑output compatibility before release.

RAG Index follows a blue‑green strategy: a new (Green) index is built in an isolated namespace, evaluated for recall, hit rate, and noise, then atomically promoted. The old (Blue) index is retained for 24 h as a rollback safety net.

Tool Schema follows SemVer. Breaking changes (e.g., optional → required) require a major version bump and contract testing.

Workflow files (YAML/JSON) are versioned in Git. Path‑coverage tests replay traces to ensure new workflow coverage matches or exceeds the previous version.

4.2 Prompt Gray Release

Traffic is split at the AI Gateway (LiteLLM, Kong AI Gateway) by user hash, sending 5% to the new Prompt while keeping model, RAG, and code constant. Metrics collected per Prompt version include QA accuracy, hallucination rate, policy violation rate, PII leakage, latency (p50/p95), token consumption, first‑contact resolution, and hand‑off rate.

Statistical significance is required: Z‑test or Bayesian A/B must show ≥95% confidence that the new Prompt improves the baseline. Future AGI’s “floor + paired CI + safety flip” rule aborts the rollout if any condition fails.

Automatic rollback cuts the Prompt’s traffic weight to zero, notifies on‑call, and clears the semantic cache to avoid ghost‑serving.

4.3 Model A/B Test and One‑Click Rollback

Model changes are registered as @challenger alongside the current @champion. In shadow mode, both versions receive the same request; only the champion’s response is returned. After sufficient samples, metrics are compared. If passed, a Canary rollout proceeds (1% → 5% → 25% → 50% → 100%) using Argo Rollouts on Kubernetes. One‑click rollback simply resets the alias to the previous version and invalidates the model cache:

def rollback_model(model_name, target_version, reason):
    registry = MLflowRegistry()
    current_champion = registry.get_alias(model_name, "champion")
    registry.set_alias(model_name, "last_failed", current_champion)
    registry.set_alias(model_name, "champion", target_version)
    gateway.invalidate_model_cache(model_name)
    cache.purge_by_model(model_name, since=current_champion.version)
    alert_oncall(reason=reason, rollback_from=current_champion, rollback_to=target_version)
    incident_log.record(model_name, current_champion, target_version, reason)

Automatic rollback triggers within seconds when any quality, safety, performance, or business metric exceeds its threshold, achieving sub‑minute recovery.

4.4 RAG Blue‑Green Deployment

The following pseudo‑code shows the full flow:

def rag_blue_green_deploy(new_doc_version, eval_queries):
    registry = VectorIndexRegistry()
    blue_id = registry.get_active_index("rag-prod")
    green_id = registry.create_index(namespace="green", doc_version=new_doc_version)
    builder = IndexBuilder(target=green_id, source_docs=new_doc_version)
    builder.run_until_complete()
    recall = eval_recall(green_id, eval_queries, k=5)
    hit_rate = eval_answer_hit(green_id, eval_queries)
    noise = eval_noise_rate(green_id, eval_queries)
    if recall < BASELINE_RECALL - 0.02 or noise > 0.10:
        registry.delete_index(green_id); raise IndexQualityGateFailed
    if not schema_compatible(blue_id, green_id):
        registry.delete_index(green_id); raise SchemaIncompatible
    registry.promote_to_active(green_id, role="rag-prod")
    gateway.swap_index_pointer("rag-prod", from_=blue_id, to=green_id)
    registry.schedule_retention(blue_id, ttl="24h", on_expire="delete")
    alert_oncall(event="rag-blue-green-switch", new=green_id, old=blue_id)

The Green index is built in isolation, validated, then atomically swapped. The Blue index remains for 24 h as a rollback fallback.

4.5 Open‑Source Tool Comparison

Key frameworks (2026 production) and their coverage:

MLflow : model lifecycle, alias‑based rollout, no native gray for Prompt.

DVC : data and pipeline versioning, no built‑in gray.

Weights & Biases : experiment tracking, enterprise gray support, eval module.

LangFuse : strong Prompt management, online eval, no native gray.

GitHub Actions : generic CI/CD, can orchestrate gray via Argo Rollouts.

Recommended stack: MLflow (model) + LangFuse (Prompt) + DVC (RAG data) + GitHub Actions (or GitLab CI) + Argo Rollouts (gray) + L7 gate (independent service). This combination is fully open‑source, clearly separates responsibilities, and is the mainstream 2026 enterprise choice.

5. Enterprise Implementation Blueprint

Define an AIReleaseBundle that records the six version numbers and digests. All releases publish a bundle rather than individual artifacts.

Three parallel sub‑pipelines run:

Code line: unit tests, lint, image build.

AI objects line: Prompt regression, model shadow, RAG evaluation, schema compatibility.

Orchestration line: workflow path‑coverage tests.

Only after all sub‑lines pass does the L7 gate decide. A successful gate creates a release tag such as release/2026-07-04-v1, which can be traced back to every object version for instant full‑bundle rollback.

5.1 Multi‑Environment Consistency

Dev, staging, and prod share the same Prompt Registry and Model Registry; environment is distinguished by labels or aliases (dev‑draft, staging‑candidate, prod‑champion). RAG index scripts are identical; only data subsets differ. Configuration changes go through GitOps (Argo CD/Flux) and PR review. Consistency SLA: staging vs prod evaluation difference ≤5%.

5.2 Rollback Strategy

Two layers:

Automatic rollback : triggered when any metric breaches its threshold (quality, safety, performance, business). AI Gateway reverts traffic within 30 s and clears caches.

One‑click rollback : on‑call presses a button to revert to the previous AIReleaseBundle. SLA ≤5 min, including cache purge and traffic switch.

5.3 Release Frequency & Change Isolation

High‑risk objects (Prompt, model, RAG index) may change only once per release; low‑risk objects (code, tool schema, workflow) can be bundled. Prompt and workflow can iterate daily; model upgrades weekly; RAG rebuilds weekly or on demand. Every change, no matter how small, must pass the L7 gate.

6. Common Pitfalls & Best Practices

Never ship a Prompt change without Golden Set regression and gray rollout.

Always keep a rollback path: keep old model alias, retain Blue RAG index for 24 h, archive Prompt versions with a drain window.

Never rebuild a RAG index in‑place; use blue‑green deployment with checkpoint support.

Isolate high‑risk changes; otherwise root‑cause analysis becomes days long.

Enforce environment parity; dev must never manually edit Prompt or model.

Gray phases must last ≥24 h to collect statistically significant data.

Always purge semantic caches after any rollback to avoid ghost‑serving.

7. Metrics & Acceptance Criteria

Hard SLA for 2026 AI production:

Version coverage: 100% of the six objects have version labels.

L7 gate coverage: 100% of releases pass through the gate.

Rollback time: <5 min for one‑click, <1 min for automatic.

Gray observation window: ≥24 h.

Environment consistency: staging vs prod metric deviation ≤5%.

Release failure rate: <5% of releases trigger rollback during gray.

MTTR: <1 min for automatic rollback scenarios.

Failure to meet any of these indicates missing governance.

8. Conclusion

Extending CI/CD from "code only" to "six‑object unified versioning" turns AI release into a disciplined, observable, and reversible process. By treating Prompt, model, and RAG index changes with the same rigor as code—using version tags, L7 evaluation, progressive gray, and instant rollback—organizations eliminate the "Friday Prompt, Monday blame" incidents and achieve reliable AI production.