Zero-Downtime Model Updates: Hot Loading & Canary Deployments for GPU Inference

1. Real Problem: Why Restarts Cause Request Failures

Assume a ranker service performing GPU forward passes for product listing requests: 80 numerical features plus context, outputting Top-N scores. Production runs 8 pods at ~600 QPS; model weights ~7 GiB; cold start, load, and representative warm-up take ~90 seconds. Models, normalization params, and label mappings update weekly.

Naive deployment proceeds: stop v41 pod → create v42 pod → download weights → load GPU → open port → receive traffic. The problem: an open HTTP port does not mean the model is ready to serve. If readiness only checks the port, v42 receives requests while still loading; if all replicas restart simultaneously, no stable version exists for quick rollback.

Verifiable goals:

Unwarmed v42 receives no business requests (evidenced by actual_revision, /readyz, mesh endpoint records).

v41 requests are not interrupted mid-flight (routing drain records; generational tests if using in-process hot loading).

v41 remains available if v42 fails (candidate failure drills; Stable endpoint stays ready).

Rollback is possible (higher revision points back to v41, or immediate routing cutback to Stable).

Every result is traceable (artifact, revision, runtime build logged in response metadata).

2. Overall Architecture: Prepare First, Then Switch Traffic

The core components are the release control plane and two independently traffic-capable service pools (Stable and Canary).

Component Responsibilities

Release Pipeline : Produces immutable artifacts, generates manifests, completes offline evaluation. Must not modify live pod models directly.

Model Registry : Stores artifact metadata, approval status, target version, audit trail. Must not proxy large inference traffic.

Release Controller : Sets Stable/Canary targets, aggregates status, promotes or rolls back. Must not load models in the request path.

Inference Runtime : Downloads, verifies, loads, dry-runs, reports actual, serves inference. Must not decide full rollout autonomously.

Service Mesh : Routes requests per approved weights. Must not judge model quality.

Dynamic loading happens inside each runtime. The runtime marks itself servable only after candidate validation succeeds; traffic switching is always decided by the control plane and routing layer.

3. Model Registry: Artifact Identity and Release Intent Must Be Separate

3.1 Artifact Is a Complete, Immutable Runtime Unit

ranker-v42

is not just weights. It carries weights, pre/post-processing resources, input contract, and compatibility constraints:

{
  "model": "ranker",
  "artifact_id": "ranker-v42",
  "runtime": "pytorch-state-dict",
  "feature_schema": "ranker-features-v3",
  "precision": "fp16",
  "files": [
    {"path": "weights.pt", "sha256": "<64-hex>", "bytes": 7516192768},
    {"path": "normalization.json", "sha256": "<64-hex>", "bytes": 8192},
    {"path": "label_mapping.json", "sha256": "<64-hex>", "bytes": 4096}
  ]
}

The downloader writes to a unique temp directory, verifies each file's size and SHA-256, then atomically publishes to a local content-addressable cache. Temp and final directories must reside on the same filesystem; crash recovery requires a completion marker, fsync, and a startup sweep policy. Digests only prove content matches a trusted manifest; the manifest must come from an authenticated, authorized registry. The downloader must also reject path traversal, unknown files, file count excess, and decompression bombs.

3.2 Rollback Relies on Revision, Not "Version Number Decrease"

artifact_id

denotes immutable content; revision denotes monotonically increasing release intent:

revision=100  artifact=ranker-v41  Stable
revision=101  artifact=ranker-v42  Canary
revision=102  artifact=ranker-v41  Rollback or re-point to verified artifact

If v42 downloads slowly and revision 102 is already published, the slow task must see the new target and abandon itself before committing. Same revision pointing to different artifacts is a control-plane consistency error and must be rejected with an alert.

3.3 PostgreSQL Metadata with CAS Publishing

Separate Stable and Canary targets to avoid ambiguous latest subscriptions:

CREATE TABLE model_artifact (
  model_name TEXT NOT NULL,
  artifact_id TEXT NOT NULL,
  manifest_uri TEXT NOT NULL,
  manifest_sha256 CHAR(64) NOT NULL,
  state TEXT NOT NULL CHECK (state IN ('pending','ready','rejected')),
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (model_name, artifact_id)
);

CREATE TABLE deployment_target (
  model_name TEXT NOT NULL,
  environment TEXT NOT NULL,
  cohort TEXT NOT NULL CHECK (cohort IN ('stable','canary')),
  revision BIGINT NOT NULL CHECK (revision > 0),
  artifact_id TEXT NOT NULL,
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (model_name, environment, cohort),
  FOREIGN KEY (model_name, artifact_id) REFERENCES model_artifact (model_name, artifact_id)
);

-- In same transaction: verify artifact is ready, insert audit record.
UPDATE deployment_target
SET artifact_id = :artifact_id,
    revision = revision + 1,
    updated_at = now()
WHERE model_name = :model_name
  AND environment = :environment
  AND cohort = :cohort
  AND revision = :expected_revision;

Callers must check affected row count. Zero rows indicate concurrent publish conflict or missing target; re-read and decide, never blind-write. Audit records must include operator, reason, old/new artifact, old/new revision, and approval info.

4. Safely Entering Canary: Download, Load, Dry Run, Ready

The runtime maintains two states: control-plane desired and successfully loaded actual. On new target arrival:

Service Mesh → Artifact Repo → Canary Runtime → Release Controller
1. desired=(rev101, ranker-v42)
2. Download manifest & artifact
3. Verify digest, schema, runtime compatibility
4. Load model & execute dry run
5. actual=(rev101, ranker-v42), ready → allow 5% traffic to Canary
6. If any step fails → keep Canary at 0%

Candidate must pass these checks before reporting actual_revision:

Verify all manifest files and cache completion marker.

Validate feature_schema, precision, model format, allowed PyTorch/CUDA/driver combos.

Construct model using service-owned architecture; never execute arbitrary Python code from artifact.

Strictly load state dict, check key layer names and parameter shapes.

Run pre-processing, forward, post-processing with production-shaped representative requests.

Verify output shape, dtype, finite values, business invariants; wait for CUDA warm-up completion.

Check peak load stays within CPU, disk, and GPU capacity budgets.

Runtime reconciliation loop skeleton (single-flight, bounded queue, bounded backoff with jitter):

from dataclasses import dataclass

@dataclass(frozen=True)
class DeploymentTarget:
    revision: int
    artifact_id: str
    manifest_uri: str
    manifest_sha256: str
    feature_schema: str

def reconcile(runtime, target: DeploymentTarget) -> None:
    """Called by dedicated background worker; not in HTTP request thread."""
    if target.revision < runtime.desired_revision:
        return  # stale control-plane notification
    runtime.remember_desired(target)

    if runtime.actual_revision == target.revision:
        return

    try:
        artifact_dir = runtime.cache.fetch_verified(
            manifest_uri=target.manifest_uri,
            manifest_sha256=target.manifest_sha256,
        )
        candidate = runtime.loader.prepare(
            artifact_dir=artifact_dir,
            expected_schema=target.feature_schema,
        )
        runtime.loader.dry_run(candidate, runtime.representative_requests)

        # Re-read desired before install to prevent v42 overwriting a newer rollback.
        if runtime.current_desired_revision() != target.revision:
            runtime.loader.dispose(candidate)
            return

        runtime.install(candidate, target)
        runtime.report_actual(target, status="ready")
    except Exception as exc:
        # Keep existing serving model; candidate failure must not mark healthy instance NotReady.
        runtime.record_candidate_failure(target, exc)
        runtime.report_actual(target, status="failed")

4.1 PyTorch Dry Run: Input Contract Matters More Than "Random Run"

def prepare_ranker_candidate(weights_path, device, representative_requests):
    model = build_ranker_v3()  # architecture owned by service code
    state = torch.load(weights_path, map_location="cpu", weights_only=True)
    model.load_state_dict(state, strict=True)
    model.to(device).eval()

    with torch.inference_mode():
        for request in representative_requests:
            inputs = move_to_device(request.model_inputs, device)
            output = model(**inputs)
            validate_ranker_output(output, request.expected_contract)

    if device.type == "cuda":
        torch.cuda.synchronize(device)  # candidate warm-up complete condition, not per-request
    return model

eval()

and inference_mode() serve different purposes. Representative requests must cover actual batch, shape, mask, sparse/missing features, and post-processing paths; random tensors cannot prove production path initialization.

5. Ready to Receive Traffic: Kubernetes Deployment & GPU Capacity

Probes must distinguish first startup, traffic-ready, and process liveness:

/startupz 200: first trusted model ready; 503: no usable model yet.

/readyz 200: has serving model and not draining; 503: no servable model or terminating.

/livez 200: process can run safely; 503: unrecoverable critical runtime error.

When candidate v42 fails but v41 is healthy, /readyz must continue returning 200; the release controller judges v42 Canary eligibility via actual, not by forcing stable instances to lose serving capability.

Example Deployment (values are illustrative; must be measured and load-tested per team SLO):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ranker-canary
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ranker
      cohort: canary
  template:
    metadata:
      labels:
        app: ranker
        cohort: canary
    spec:
      nodeSelector:
        workload: inference-gpu
      terminationGracePeriodSeconds: 120
      containers:
      - name: ranker
        image: registry.example/ranker:runtime-20261008
        ports:
        - containerPort: 8080
        startupProbe:
          httpGet:
            path: /startupz
            port: 8080
          periodSeconds: 5
          failureThreshold: 120
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          periodSeconds: 5
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /livez
            port: 8080
          periodSeconds: 15
          failureThreshold: 3
        resources:
          requests:
            cpu: "2"
            memory: 8Gi
          limits:
            memory: 12Gi
            nvidia.com/gpu: 1

GPU peak memory = weights + live inference + load temporary + warmup workspace + runtime + fragmentation margin. Example: 24 GiB GPU, 7 GiB weights, 4 GiB in-flight + runtime, 3 GiB load/warmup temp, 3 GiB margin ≈ 17 GiB. If process holds both old 7 GiB and new 8 GiB simultaneously, may hit 25 GiB — cannot double-buffer. torch.cuda.empty_cache() only releases allocator's unused cached blocks, not tensors still referenced.

GPU inference autoscaling should not rely solely on CPU. Combine queue length, concurrency, P95/P99, GPU utilization, and memory watermark. HPA manages replica count; cluster autoscaler manages schedulable GPUs. Model download + 90s warm-up means scale-out latency; Canary releases must reserve extra GPU capacity.

6. Canary Rollout: 0% to 100%, Every Step Has Stop Conditions

Stable and Canary keep independent targets:

Stable: revision=100, artifact=ranker-v41
Canary: revision=101, artifact=ranker-v42

Only after at least one Canary instance reports actual=(101, ranker-v42) and ready capacity is sufficient does routing begin. Istio 95/5 stage example:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: ranker
spec:
  host: ranker
  subsets:
  - name: stable
    labels:
      cohort: stable
  - name: canary
    labels:
      cohort: canary
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ranker
spec:
  hosts: [ranker]
  http:
  - route:
    - destination:
        host: ranker
        subset: stable
      weight: 95
    - destination:
        host: ranker
        subset: canary
      weight: 5
    timeout: 2s

Actionable promotion table (timing and thresholds from team SLO, not copied blindly):

Warm-up : 0% traffic. Prereq: all Canary verification, dry run, ready. Stop if any candidate verification/warm-up fails.

Small traffic : 5%. Prereq: capacity, probes, actual consistent. Stop/rollback on error rate, P99, GPU memory, or queue anomalies.

Expand : 25% / 50%. Prereq: metrics healthy, sufficient sample size. Stop/rollback on technical or business metric degradation.

Full : 100%. Prereq: observation window meets thresholds. Immediate routing cutback to Stable on issues.

Retire : 100% new version. Prereq: old requests drained, observation period ended. Retain old artifact reload capability.

5% requests ≠ 5% GPU consumption: input length, batch, tenant, cache hits alter resource proportion. User-level A/B requires stable bucketing, not just random weight routing.

Rollback has two paths: if old Stable still ready, cut new request routing weight back to Stable; if old pool already recycled, publish higher revision pointing to old artifact, re-warm, then cut traffic. Neither guarantees unconditional "second-level rollback".

7. Multi-Instance Convergence, Draining & Failure Handling

Control plane stores desired; data plane reports actual. Instances report at least: instance ID, cohort, desired revision, actual revision, artifact, ready state, last error, heartbeat, servable capacity.

Control plane disconnect: verified models continue serving with alert; reject new artifacts not from trusted registry; new pod startup from trusted local cache defined by pre-set recovery policy. On reconnect, re-converge to authoritative desired; same revision with different artifact is a severe consistency error — never auto-guess.

On scale-down, recycle, or SIGTERM: service must first refuse new admissions, wait for endpoint propagation, drain in-flight requests, then release resources. terminationGracePeriodSeconds must cover propagation and drain budget; preStop (if present) consumes same grace period.

For single-pass inference, drain after routing removal. For streaming LLM, old sessions must stay bound to old model; KV Cache is tied to weights, adapter, positional encoding, and runtime layout — cannot be reused across models just because tokenizer matches.

8. Advanced Optimization: In-Process Generational Hot Loading

Cross-pod Canary already achieves zero-downtime. Only consider in-process model switching when update frequency is high, cold start is a true bottleneck, GPU has headroom for double-buffering old and new models, and team has mature observability.

Switch unit must be a full generation: model, artifact, revision, feature schema, normalization params, label mapping, and runtime build together. On request entry, increment generation refcount; on new generation publish, retire old generation; after old generation refcount reaches zero and real GPU work completes, release asynchronously.

Core rules: borrow, read version, and increment refcount under same short lock; lock does not cover download and inference; cleanup runs outside lock by observable worker; CUDA non-default streams, async D2H copies, or background post-processing must use corresponding CUDA events as release conditions, not HTTP return, coroutine cancellation, or Python function return.

Runnable Python 3.10+ demo (no PyTorch dependency) verifying concurrency semantics: old requests pinned to v41, new requests see v42, stale candidate cannot overwrite rollback, multiple concurrent releases queue cleanup once, cleanup failure does not affect successful responses. Save as ranker_generation_demo.py and run python3 ranker_generation_demo.py.

from __future__ import annotations
from contextlib import contextmanager
from dataclasses import dataclass
from queue import SimpleQueue
from threading import Barrier, Event, Lock, Thread
from typing import Iterator, Optional

class ToyRanker:
    def __init__(self, bias: int) -> None:
        self.bias = bias
        self.closed = False
    def score(self, value: int) -> int:
        if self.closed:
            raise RuntimeError("ranker already closed")
        return value + self.bias
    def close(self) -> None:
        self.closed = True

@dataclass(eq=False)
class Generation:
    revision: int
    artifact_id: str
    model: ToyRanker
    refs: int = 0
    retired: bool = False
    cleanup_queued: bool = False

class GenerationHolder:
    def __init__(self) -> None:
        self._lock = Lock()
        self._current: Optional[Generation] = None
        self._desired = (0, "")
        self._cleanup_queue: SimpleQueue[Generation] = SimpleQueue()
    def set_desired(self, revision: int, artifact_id: str) -> None:
        with self._lock:
            known_revision, known_artifact = self._desired
            if revision < known_revision:
                return
            if revision == known_revision:
                if artifact_id != known_artifact:
                    raise ValueError("same revision has different artifact")
                return
            self._desired = (revision, artifact_id)
    def _queue_cleanup_locked(self, generation: Generation) -> None:
        if not generation.cleanup_queued:
            generation.cleanup_queued = True
            self._cleanup_queue.put(generation)
    @contextmanager
    def borrow(self) -> Iterator[Generation]:
        with self._lock:
            generation = self._current
            if generation is None:
                raise RuntimeError("ranker is not ready")
            generation.refs += 1
        try:
            yield generation
        finally:
            with self._lock:
                generation.refs -= 1
                if generation.refs < 0:
                    raise AssertionError("unbalanced generation release")
                if generation.retired and generation.refs == 0:
                    self._queue_cleanup_locked(generation)
    def install(self, candidate: Generation) -> bool:
        # Caller owns candidate exclusively, and has completed real model load and dry run.
        with self._lock:
            current = self._current
            accepted = (
                (candidate.revision, candidate.artifact_id) == self._desired
                and (current is None or candidate.revision > current.revision)
            )
            if not accepted:
                self._queue_cleanup_locked(candidate)
                return False
            self._current = candidate
            if current is not None:
                current.retired = True
                if current.refs == 0:
                    self._queue_cleanup_locked(current)
            return True
    def drain_cleanup(self) -> None:
        # In production, replace with observable, retryable background worker; exceptions must not propagate to requests.
        while not self._cleanup_queue.empty():
            generation = self._cleanup_queue.get()
            try:
                generation.model.close()
            except Exception as exc:
                print(f"cleanup failed for {generation.artifact_id}: {exc}")

def main() -> None:
    holder = GenerationHolder()
    v41 = Generation(100, "ranker-v41", ToyRanker(41))
    holder.set_desired(100, "ranker-v41")
    assert holder.install(v41)
    # Old request borrowed v41; main thread publishes v42; old request allowed to finish last.
    borrowed = Barrier(2)
    allow_old_request_to_finish = Event()
    old_result: list[int] = []
    def old_request() -> None:
        with holder.borrow() as generation:
            assert generation.artifact_id == "ranker-v41"
            borrowed.wait()
            allow_old_request_to_finish.wait(timeout=2)
            old_result.append(generation.model.score(1))
    thread = Thread(target=old_request)
    thread.start()
    borrowed.wait(timeout=2)
    holder.set_desired(101, "ranker-v42")
    v42 = Generation(101, "ranker-v42", ToyRanker(42))
    assert holder.install(v42)
    with holder.borrow() as fresh:
        assert (fresh.artifact_id, fresh.model.score(1)) == ("ranker-v42", 43)
    assert not v41.model.closed
    allow_old_request_to_finish.set()
    thread.join(timeout=2)
    assert not thread.is_alive()
    assert old_result == [42]
    holder.drain_cleanup()
    assert v41.model.closed
    # Rollback uses higher revision; stale v42 candidate does not overwrite new target.
    holder.set_desired(102, "ranker-v41")
    stale_v42 = Generation(101, "ranker-v42", ToyRanker(42))
    assert not holder.install(stale_v42)
    v41_rollback = Generation(102, "ranker-v41", ToyRanker(41))
    assert holder.install(v41_rollback)
    holder.drain_cleanup()
    assert stale_v42.model.closed
    with holder.borrow() as current:
        assert (current.revision, current.artifact_id) == (102, "ranker-v41")
    print("PASS: pinning, concurrent switch, stale rejection, rollback")

if __name__ == "__main__":
    main()

Suggested module boundaries for real projects:

ranker-service/
├── app.py                 # /predict, /startupz, /readyz, /livez, graceful shutdown
├── registry_client.py     # desired target pull, actual status push
├── artifact_cache.py      # temp dir download, manifest verify, atomic cache publish
├── model_loader.py        # PyTorch build, strict load, dry run, CUDA events
├── generation_holder.py   # GenerationHolder from this article
├── rollout_metrics.py     # Prometheus metrics and audit fields
└── tests/
    ├── test_artifact_cache.py
    ├── test_reconcile.py
    └── test_generation_switch.py

Minimum automated verification: old requests cross switch still use v41; new requests only see v42; slow v42 load cannot overwrite revision 102 rollback; multiple simultaneous request completions schedule cleanup once; cleanup failure does not alter already successful responses. This optimization complements, not replaces, Stable/Canary routing, reducing per-pod cold-start cost.

9. Production Acceptance: Prove "Seamless" with Evidence

Define acceptance gates before release. Minimum observability: Stable/Canary QPS, error rate, P95/P99; queue and reject counts; download/load/dry-run/commit latency; desired/actual mismatch; GPU allocated/reserved vs device usage; per artifact, revision, runtime build; output quality or business proxy metrics.

Fault Injection Tests

Artifact truncated, digest or schema mismatch → Canary not ready, Stable continues serving.

Rollback issued during slow v42 load → Late candidate cannot overwrite higher revision.

Dry run passes but Canary P99 degrades → Stop promotion, route back to Stable.

Canary GPU OOM → Instance drained, Stable capacity maintains service.

Control plane disconnect or duplicate notifications → Verified models keep serving, re-converge on recovery.

SIGTERM coincides with traffic shift → Stop new admissions, drain, shutdown, no re-admission.

The real deliverable is not a "hot loading switch" but an evidence chain: artifact trusted, candidate warmed, unready instances receive no traffic, traffic switching controllable, old version can absorb rollback, every result traceable. Achieve these, and model services transition smoothly to new versions without downtime.

References

PyTorch CUDA semantics: async execution, streams, synchronization.

PyTorch empty_cache: cache allocator behavior boundaries.

PyTorch inference_mode: thread-local inference context.

Kubernetes Probes: startup, readiness, liveness responsibilities.

Istio Traffic Shifting: weight-based traffic migration.