Zero-Downtime Model Updates: Hot Loading & Canary Deployments for GPU Inference
This article details a zero-downtime model update strategy using immutable artifacts, a model registry, canary deployments with readiness probes, traffic shifting via service mesh, and optional in-process generational hot loading, demonstrated with a GPU ranking service handling 600 QPS and 7 GiB models.
