High Availability for LLM Inference: Health Checks, Warmup, Failover & Degradation

Introduction: Three Real Incidents

The article opens with three production incidents that expose the gap between naive Kubernetes deployments and reliable LLM inference:

Rolling upgrade 100% errors : A single-replica vLLM Deployment used a readinessProbe hitting /health (returns 200 once HTTP port is up). During upgrade, the new Pod passed readiness while still loading 30 GB weights (~40s), causing all requests to hit model not loaded 500 errors.

Node restart "fake healthy" : After a GPU node reboot, /health returned 200 but all requests timed out. GPU XID errors and NCCL hangs were invisible to the HTTP-only probe.

OOM restart loop : A long-context request triggered OOM; restartPolicy: Always restarted the Pod, which spent 8 minutes reloading weights, then hit the same request again — 11 restarts in one night.

Root cause: health checks verified "process alive" not "can serve inference". This stems from applying stateless-web assumptions to stateful, GPU-bound, slow-start inference workloads.

Why Inference HA Differs from Web Services

A comparison highlights seven broken assumptions:

Startup time : Web 1–5 s vs Inference 2–15 min (model size / storage bandwidth)

Resources : Web CPU/RAM overcommit OK vs Inference GPU VRAM exclusive, no overcommit

State : Web stateless vs Inference KV cache stateful, requests interfere

Scale-out : Web immediate capacity vs Inference wait for model load

Failure mode : Web process crash → 5xx vs Inference process alive but GPU dead → timeout

Connections : Web short-lived vs Inference streaming long-lived (seconds to minutes)

Single-instance capacity : Web thousands QPS vs Inference dozens to hundreds QPS

Four design principles follow:

Ready ≠ port open — ready must mean "can complete a real inference".

Treat startup separately — use startupProbe with generous timeout, not initialDelaySeconds.

Plan capacity for worst case — with 3 replicas, losing one must leave 2 handling full peak traffic.

Degradation is part of design — GPU failure, load failure, OOM are inevitable; system must have "functional but degraded" paths.

Applicability Tiers

Four architecture levels (L1–L4) match availability targets (95% → 99.9%+). Most enterprises need L2–L3: multi-replica + layered health checks + warmup + gateway failover + model fallback. The article targets L3.

Environment & Version Assumptions

Engine: vLLM (OpenAI-compatible), version via vllm --version; parameters based on vllm serve (check --help per version).

Orchestration: Kubernetes 1.28+, containerd, NVIDIA GPU Operator.

GPUs: Mixed A100 80G / L20 / 4090.

Storage: Model weights on NFS or object storage + local NVMe cache.

Gateway: Nginx/Ingress or Higress/Envoy with active health checks.

Monitoring: Prometheus + node_exporter + DCGM exporter + vLLM /metrics.

Models: 7B–72B, tensor parallel (TP=2/4).

Three version pitfalls: (1) vLLM metric names change across versions (e.g., vllm:gpu_cache_usage_perc vs vllm:kv_cache_usage_perc) — always curl /metrics to verify; (2) --max-model-len default comes from model config.json, handling of over-length requests varies; (3) startupProbe timeout budget must cover slowest cold load (72B from network storage >15 min).

Core Knowledge: Startup Lifecycle & Probes

5.1 vLLM Startup Timeline

t=0      container start, Python process up
t=1s     import torch/vllm, init CUDA context
t=5s     start loading weights (safetensors → GPU)
t=5s–X   weight load (7B/local ~40s, 72B/network ~8–15 min)
t=X      allocate KV cache (per gpu_memory_utilization)
t=X+5s   CUDA graph capture (kernel compile, tens of seconds)
t=X+40s  HTTP server listening
t=X+41s  /health returns 200
t=X+42s  first real inference (slow if no warmup)

Key: /health returns 200 at t=X+41s but true readiness is t=X+42s+. The 1-second window becomes a flood of timeouts during rolling upgrades. CUDA graph capture (skipped with --enforce-eager, but 10–30% slower inference) is why warmup exists.

5.2 Three Probe Semantics

startupProbe : failure restarts container; should check model loaded; typical config: failure threshold high (60×10s = 10 min)

readinessProbe : failure removes from Service, no restart; should check can do real inference; typical config: period 10s, failure threshold 2–3

livenessProbe : failure restarts container; should check process not deadlocked; typical config: period 30s, failure threshold 3–5

Three rules: (1) livenessProbe must coexist with startupProbe (otherwise liveness kills during load); (2) livenessProbe must not check dependencies (else upstream blip restarts all pods); (3) readinessProbe must check "can infer", not "port open".

5.3 Why /health Is Insufficient

vLLM's /health only checks HTTP server. It does NOT verify: GPU health (XID/ECC errors, GPU fallback), KV cache allocatability (VRAM fragmentation), upstream TP peers. Correct readiness = fire a minimal real inference request:

curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model":"qwen","max_tokens":1,"messages":[{"role":"user","content":"hi"}]}'

Cost: each probe consumes ~tens of ms GPU. Mitigate: period 10–15s, max_tokens:1, never use in liveness. Compromise: lightweight /health + "recent successful inference" metric.

5.4 Five Warmup Levels

L1 : Model weights into page cache — solves cold start disk I/O — method: cat weight files once

L2 : CUDA graph / kernel compile — solves first request very slow — method: send warmup requests at startup

L3 : KV cache VRAM pool — solves first allocation jitter — vLLM pre-allocates at start (usually no extra step)

L4 : Gateway routing table — solves new Pod not receiving traffic — method: add to endpoints only after readiness

L5 : Client pool / DNS — solves first connect slow, DNS timeout — method: client warmup, keepalive

Most teams only do L2, ignoring L1/L4 → rolling upgrades still jitter. L1 detail: use node-local NVMe cache (DaemonSet pre-pull from object storage/NFS) + cat to warm page cache. L4 detail: preStop: sleep 15 lets old Pod stay alive while endpoints update.

Production Deployment Manifest (L3 Target)

Complete YAML for 3-replica Deployment (A100 80G×2, TP=2, 32B model) with explanations: maxSurge:1, maxUnavailable:0 — mandatory: start new, wait ready, then kill old. Requires spare GPU; no alternative because VRAM cannot overcommit. terminationGracePeriodSeconds:120 — default 30s too short for streaming requests (e.g., 4096 tokens @ 30 tok/s = 136s). Set ≥ max generation time. preStop: sleep 15 — solves ready/terminate race; waits for gateway to drain. startupProbe + livenessProbe coexistence — startup gives load time, liveness catches runtime hangs. readinessProbe using /health (acceptable but not ideal). Ideal: exec probe running minimal inference curl (shown). Trade-off: exec forks + real inference; use for core biz, httpGet for internal tools. /dev/shm mount (8Gi emptyDir:Memory) — required for TP>1 (NCCL shared memory); default 64MB causes Bus error. HF_HUB_OFFLINE=1 — prevents startup hang on network model-check. hostPath for model cache (risky: node-level mutation). Better: CSI local PV or initContainer sync to emptyDir. If using hostPath, readOnly:true mandatory. PodDisruptionBudget minAvailable:2 — blocks kubectl drain until 2 replicas ready; only guarantee for planned maintenance.

Deploy & Verify

kubectl apply -n llm -f vllm-infer.yaml
kubectl -n llm get pods -l app=vllm-infer -w
# Watch for READY 1/1 after ~7 min
kubectl -n llm logs deploy/vllm-infer --tail=50 | grep -E 'init engine|Starting|Application startup|route'
# Key log order: "Model loading took..." → "init engine..." → "Starting vLLM API server" → "Application startup complete"
# Verify real inference:
kubectl -n llm run curl-test --rm -it --image=curlimages/curl:8.5.0 --restart=Never -- \
  curl -s -X POST http://vllm-infer.llm.svc.cluster.local:8000/v1/chat/completions ...

Warmup: From "Starts" to "Handles Load"

7.1 Weight Preload (L1)

DaemonSet per GPU node syncs model to local NVMe via initContainer (example uses aws s3 sync with IRSA; on-prem use NFS/MinIO). Container runs sleep infinity; purpose is node-level cache. Optional:

find ... -name '*.safetensors' -print0 | xargs -0 -n1 -P4 cat > /dev/null

to warm page cache (ephemeral, only useful right before Pod start).

7.2 Inference Warmup (L2)

Goal: trigger CUDA graph capture, kernel compile, KV cache alloc. Best done by external Job after Pod ready (not postStart which runs concurrently). Example Job sends: 3× short (8 tokens), 1× medium (256 tokens), 8× concurrent short — covering different batch/seq lengths for graph capture. Design points: use real model name, cover lengths, concurrent via & wait, total <1 min.

7.3 Gateway Warmup (L4)

Endpoints propagation delay (1–5s). Measure: Pod Ready timestamp vs first gateway log hit. Mitigations: preStop sleep (done), gateway active health checks (2–3s interval), adequate terminationGracePeriodSeconds.

7.4 Warmup Validation

Restart Pod, immediately after readiness fire first request, measure time_starttransfer (TTFT). Typical: unwarmed 3–8s (kernel compile), warmed 200–400ms. If still slow: warmup Job ran on old Pod, wrong model name, or --enforce-eager enabled.

Failover: Detect → Switch

8.1 Three-Layer Health Checks

Process : K8s readinessProbe — detection 10–30s — covers process hang, port down

Gateway : Active HTTP probes — detection 2–10s — covers 5xx, timeout (catches probe misconfig)

Business : Success rate / latency alerts — detection 1–5 min — covers 200 OK but wrong content, latency degradation

Nginx passive: max_fails=2 fail_timeout=10s (10s window, 2 fails → mark down 10s). Envoy active + outlierDetection (consecutive 5xx, ejection time, max_ejection_percent:50 guardrail).

8.2 Failover Modes

Multi-replica + LB (most common): 3 replicas, lose 1 → 67% capacity. Requires single-replica capacity ×2 ≥ peak.

Active-passive: standby idle (still occupies GPU) or cold (15 min spin-up). Only for "can tolerate 15 min downtime".

Cross-model fallback: primary down → route to smaller/cheaper model. This is degradation, not HA.

L3 combines mode 1 + 3.

8.3 Cooldown & Half-Open

Recovered instance must not receive full traffic instantly (risk re-crash). Pattern: fail → cool 30s → half-open (10% traffic probe) → success → full restore; fail → double cool (max 5 min). Nginx only does cooldown; Envoy outlierDetection approximates; custom gateway can implement explicit half-open.

8.4 Drills (Mandatory)

Record RTO: detection time (<30s), switch time (<10s), capacity loss (single failure ≥67%). Drills: kill pod (observe error rate, queue), kill -STOP 1 (simulate fake-alive), GPU reset (high risk, dedicated test node only).

Degradation: Turn "Unavailable" into "Slightly Worse"

9.1 Five Degradation Levels

D1 : Semantic cache hit (exact + embedding similarity ≥0.95) — near zero impact — gateway/app

D2 : Parameter degrade (half max_tokens, disable streaming) — shorter answers, longer wait — gateway/app

D3 : Model fallback (large → small) — quality drop — gateway (e.g., LiteLLM router fallbacks)

D4 : Queue with timeout (429 + Retry-After) — wait required — gateway/app

D5 : Disable AI feature, fall back to rule/retrieval — feature missing, system up — app

D1 cache: exact match (Redis key = hash(model+messages+temp)) → embedding similarity (threshold 0.95; lower causes wrong answers). D3 direction: fallbacks = failure degrade (large→small), context_window_fallbacks = capability upgrade (small→large) — reversing causes long-context requests to hit short-context model. D4 queue better than immediate 429; vLLM internal queue ( --max-num-seqs) lacks timeout, so gateway must enforce request-level timeout.

9.2 Automated Triggers (Multi-Metric)

Queue length: num_requests_waiting > 50 for 60s → D2 param degrade

KV cache usage: > 0.95 for 30s → D3 model fallback

P95 TTFT: > 5s for 2 min → D3 model fallback

Error rate: > 5% for 1 min → switch backup model

All backends down: health checks all fail → D5 feature off

Avoid single-metric triggers; add time persistence to filter spikes.

9.3 Degradation Switch Implementation

Switch must be highly available — not in a service that can fail. Simplest: config center (Nacos/Apollo/Consul) + local cache with TTL. Critical: on config center failure, retain last known value (do NOT reset to "no degradation").

Metrics & Alerting: Observe to Control

10.1 vLLM Metrics (verify names per version)

Key: num_requests_waiting (most important overload signal), num_requests_running, gpu_cache_usage_perc (>0.9 = VRAM pressure), cpu_cache_usage_perc (>0 = swapping, perf cliff), time_to_first_token_seconds (user-perceived latency), time_per_output_token_seconds, e2e_request_latency_seconds, request_success_total (by finish_reason), prompt_tokens_total, generation_tokens_total, request_prompt_tokens (detect long-context surge).

10.2 GPU Metrics (DCGM Exporter)

Critical: DCGM_FI_DEV_XID_ERRORS (>0 immediate alert). XID codes: 48 (double-bit ECC → replace GPU), 79 (GPU off bus → reboot node), 13/31/43 (app/driver bug → restart process), 63 (page retirement → capacity loss), 94/95 (correctable/uncorrectable ECC). Also monitor GPU util, mem copy util, VRAM used, temp (>85°C), power violation, NVLink CRC errors.

10.3 Alert Rules (Prometheus)

Examples: VLLMQueueBacklog (waiting>20 for 3m), VLLMKVCacheHigh (usage>0.92 for 5m), VLLMTTFTSlow (P95 TTFT>3s for 5m), GPUXIDError (increase>0 for 1m, severity critical), VLLMNoTraffic (0 success req for 10m — detects fake-alive). Thresholds must be baselined: run 2 weeks, set alert at 2× P95.

Troubleshooting: Six Common Patterns

11.1 502/503/504 Discrimination

502 : Gateway cannot reach backend — first check backend process, port, Service endpoints

503 : No healthy backend — first check all replicas unready (readiness failing)

504 : Backend timeout — first check queue, long generation, timeout misconfig

Hidden 503 cause: exec readiness with wrong API key/model name → permanent 401/404 → pod never ready.

11.2 TTFT Spike (GPU util stable, error rate stable)

Queue: check num_requests_waiting → scale or degrade.

Prefix cache invalidation: system prompt change (even whitespace) busts cache; monitor gpu_prefix_cache_hit_rate.

Long prompt contention: 30K token prefill starves short requests; check request_prompt_tokens distribution; mitigate with chunked prefill ( --max-num-batched-tokens).

GPU throttling: nvidia-smi --query-gpu=clocks_throttle_reasons.active non-zero → thermal/power limit.

11.3 Throughput Drop, Latency Normal

Usually effective batch size shrank. Causes: arrival rate down (not a fault), max_num_seqs lowered, KV cache full ( gpu_cache_usage_perc ≈1), request length shift (many short requests). Check avg_generation_throughput_toks_per_s — if down while GPU util flat, it's scheduling not compute.

11.4 NCCL Timeout (TP>1 / PP>1)

Three causes: network (IB/RoCE, MTU, wrong NIC — set NCCL_SOCKET_IFNAME), GPU (XID, NVLink degrade — check nvidia-smi nvlink -s, dmesg -T | grep -i xid), resource ( /dev/shm too small — verify 8Gi via df -h /dev/shm). Principle: don't just restart; inspect XID/dmesg to distinguish hardware vs software.

11.5 VRAM Leak / Gradual OOM

Symptom: OOM after hours, restart fixes temporarily. Causes: (1) max_model_len too large (e.g., 128K vs actual 4K) → KV cache fragmentation; fix: right-size or enable chunked prefill. (2) Multiple vLLM processes — check nvidia-smi --query-compute-apps. (3) Driver/framework leak — linear DCGM_FI_DEV_FB_USED rise indicates leak.

11.6 Rolling Upgrade Stuck

Top causes: (1) No spare GPU for maxSurge:1 (events show Insufficient nvidia.com/gpu). (2) startupProbe timeout too short (model load slower). (3) Readiness failing (bad API key/model name in exec probe). (4) PDB blocking ( minAvailable:2 with only 2 healthy). Fix root cause; don't force-delete pods (kills multiple replicas).

Risks & Rollback

12.1 High-Risk Operations

Change gpu_memory_utilization — impact: all requests on pod — pre-check: confirm no OOM — rollback: revert + restart

Change max_model_len — impact: all requests — pre-check: verify business max context — rollback: revert + restart

Reduce replicas — impact: cluster capacity — pre-check: remaining > peak — rollback: scale up

Upgrade vLLM image — impact: all — pre-check: test env regression, pin digest — rollback: rollback digest

Change max_num_seqs — impact: concurrency — pre-check: load test — rollback: revert

GPU reset / node reboot — impact: whole node — pre-check: confirm no other pods — rollback: reschedule (no instant rollback)

12.2 Rollback Steps

Config rollback: backup Deployment YAML; prefer ConfigMap for args (change ConfigMap + rollout restart — but remember ConfigMap change doesn't auto-restart). Image rollback: rollout history + rollout undo --to-revision=N (requires --record or ReplicaSet history). Post-rollback verify: (1) replica count restored, (2) real inference works, (3) metrics normalize (wait 5 min).

Production Checklist

Capacity : single-replica QPS ×0.6; 3 replicas lose 1 → remaining 2 ≥ peak; spare GPU for maxSurge:1.

Change Management : model version, vLLM version, key params ( max_model_len, gpu_memory_utilization, max_num_seqs) via change request; backup YAML; off-peak window; rollback plan + verification script; canary: test → 1 replica → full (30 min each).

Permissions : restrict kubectl delete pod in llm ns (RBAC); GPU node hardware ops (e.g., nvidia-smi -r) only by ops with confirmation of no business pods; API keys in Secrets (not args visible in ps).

Monitoring Baseline : 2 weeks pre-prod collecting TTFT, e2e, throughput, queue P50/P95/P99; alerts based on baseline, not copied numbers; business-level monitoring (success rate, content sampling).

Drills : quarterly (kill replica, fake-alive, GPU fault); record RTO; verify degradation switches; low-traffic window, pre-announced.

Cost : gpu_memory_utilization 0.85–0.90 sweet spot (0.95 raises OOM/fragmentation); off-peak scale-down possible but re-warm costs 5–15 min.

Summary

The three opening incidents resolve to:

Rolling upgrade errors → startupProbe budget + maxUnavailable:0 + preStop + warmup. Core: ready ≠ port open.

Fake-alive after node reboot → health checks must cover GPU layer (DCGM + XID). Core: process alive ≠ can infer.

OOM restart loop → limit max_model_len, add StartLimitBurst, design degradation paths. Core: restart ≠ fix, only delays fault.

Four takeaways: (1) Inference HA is engineering compromise under "slow start" and "exclusive VRAM" constraints. (2) Health checks must be layered and test capability, not existence. (3) Degradation must be pre-designed and rehearsed; degradation switches themselves must be HA. (4) All thresholds come from your baseline, not documentation.