Production LLM Inference with vLLM/TGI: Deployment, Tuning & Observability Guide

Background & Challenges

LLM inference differs fundamentally from traditional web services: it is doubly constrained by GPU memory bandwidth and capacity. Deploying billion-parameter models (Qwen2.5, Llama 3, DeepSeek) in production faces four core challenges: extremely low GPU memory utilization due to static KV cache allocation; static batching inefficiency where long sequences block entire batches; trade-offs between time-to-first-token (TTFT) and inter-token latency (ITL); and avalanche effects under burst traffic causing CUDA OOM crashes.

Core Architecture & Mechanisms

PagedAttention

vLLM adopts OS-style paging: KV cache is split into fixed-size blocks (default 16/32 tokens) managed via a block table mapping logical to physical blocks. This eliminates fragmentation (internal fragmentation <4%) and enables zero-copy sharing of common prefixes across requests.

Continuous Batching

Unlike static batching, continuous batching operates at iteration level. When a sequence finishes, its blocks are immediately released and new waiting requests are pulled in, keeping Tensor Cores saturated and boosting throughput 2-4x.

Chunked Prefill

Long prompts are split into chunks (e.g., 512/1024 tokens) and mixed with decode requests in each scheduling iteration, flattening ITL spikes by >80%. Chunk size must balance context-switch overhead.

Automatic Prefix Caching (APC)

APC uses a hash tree to detect and reuse cached prefix blocks (system prompts, conversation history), skipping prefill matrix multiplications. This cuts TTFT from hundreds of ms to tens of ms and reduces memory/compute by >50%.

vLLM V1 Engine & Compilation Levels

V1 decouples scheduling from model forward pass and introduces optimization levels -O0 to -O3. -O3 (production default) integrates FlashAttention-3/FlashInfer, maximizes CUDA Graph capture and kernel fusion, requiring warm-up but delivering optimal steady-state throughput.

Deployment Planning & Hardware Topology

VRAM Estimation Formula

Total VRAM = Model Weights + KV Cache + Activations. Weights: P × B × 1.2 (P=params, B=bytes per param). KV Cache per token: 2 × L × H_kv × D_head × B_dtype. Activations: 2-6 GB. Example: Qwen2.5-72B BF16 needs ~173 GB (4×80GB GPUs).

Hardware Selection Table

Provides concrete GPU configurations for 7B/14B/32B/70B/72B and MoE models, including quantization (BF16, FP8, AWQ), parallelism (TP=1/2/4/8), and estimated max concurrency at 8K context.

Parallelism Principles

Tensor Parallelism (TP) requires intra-node NVLink/NVSwitch (≥600 GB/s); cross-node TP kills throughput. Pipeline Parallelism (PP) suits cross-node with 200-400 Gbps InfiniBand/RoCE.

Preflight Checks & Container Hardening

Host-level validation script checks driver version (≥550.54.14, CUDA ≥12.4), NVLink topology (must show NV# for TP), GPU persistence mode, /dev/shm size (≥50% RAM), and CPU governor (performance). Docker/containerd must pass GPU passthrough test; fix with nvidia-ctk runtime configure.

Single-Node Docker Deployment

Production run script for 4×GPU Qwen2.5-72B includes: --ipc=host, --shm-size=32g, ulimits, -v model mount, NCCL_DEBUG=WARN, NCCL_IB_DISABLE=1, and key vLLM args: --tensor-parallel-size 4, --max-model-len 8192, --gpu-memory-utilization 0.92, --max-num-seqs 128, --block-size 32, --enable-prefix-caching, --dtype bfloat16, -O3. Health probe polls /health endpoint and runs a test inference.

Kubernetes HA Deployment

Deployment YAML with pod anti-affinity, emptyDir medium:Memory for /dev/shm (32Gi), resource requests/limits (4 GPUs, 16-32 CPU, 128-256Gi RAM), startupProbe (90s initialDelay, 30 failures = 390s max), liveness/readiness probes. Service uses ClusterIP. RollingUpdate strategy with maxSurge=1, maxUnavailable=0.

Core Parameter Tuning & Decision Matrix

--gpu-memory-utilization : default 0.90; production 0.92-0.95. With -O3 CUDA Graph, reserve 2-4GB → 0.92. >0.96 risks OOM.

--max-num-seqs : default 256; production 64-128. Limits concurrent sequences per iteration; lower values improve P99 latency stability for large models.

--block-size : 16 or 32; 32 recommended on Ampere/Hopper for better FlashAttention memory alignment (~5-8% throughput gain).

--kv-cache-dtype fp8 : on Ada/Hopper halves KV cache memory, doubles concurrency, perplexity loss <0.1%.

Scenario-based combos: RAG (throughput priority) uses max-model-len 16384, max-num-seqs 64; code assistant (TTFT sensitive) uses max-model-len 8192, max-num-seqs 32, -O3; offline summarization uses max-model-len 32768, max-num-seqs 128, fp8 KV.

TGI Deployment & Migration to vLLM

TGI 3.3.7 (maintenance mode) launch with --max-batch-prefill-tokens, --max-batch-total-tokens, --cuda-memory-fraction 0.92. Migration path: decouple clients to OpenAI-compatible SDK, canary 5% traffic via gateway, align Prometheus metrics. vLLM advantages: active model support, industrial-grade APC, native OpenAI API, PagedAttention memory management.

Observability: Metrics & Alerting

Key vLLM Prometheus metrics: vllm:num_requests_waiting (queue depth), vllm:num_requests_running, vllm:gpu_cache_usage_factor (KV cache usage), vllm:time_to_first_token_seconds (TTFT), vllm:time_per_output_token_seconds (ITL), vllm:num_preemptions_total (preemption counter). Alert rules: preemption >0 (critical), queue >15 (warning), TTFT P95 >2.5s (warning), KV cache usage >0.95 (critical).

KEDA Autoscaling

Uses vllm:num_requests_waiting average across replicas as scaling trigger. ScaledObject with minReplicas=2, maxReplicas=6, pollingInterval=15s, cooldown=300s. Scale-up stabilization 0s, scale-down 300s. Threshold: average queue >5 per replica.

Benchmarking Methodology

Rejects ab/HTTP tools; uses GuideLLM or custom async client (Python aiohttp) measuring TTFT, ITL, token throughput via SSE streaming. Test dataset mixes short/long prompts per production length distribution (P50/P90/P99). Reports P50/P95/P99 TTFT, total tokens/s, RPS.

Troubleshooting Common Failures

Frequent Preemption (SWAP/RECOMPUTE) : gpu_cache_usage_factor stuck at 1.0, num_preemptions_total rising. Fix: reduce max-num-seqs, lower max-model-len, increase gpu-memory-utilization or enable fp8 KV, scale out.

NCCL Deadlock : container sees multiple CNI interfaces (flannel, calico); NCCL picks wrong interface. Fix: export NCCL_IB_DISABLE=1, NCCL_SOCKET_IFNAME=lo (single-node) or explicit IB interface (cross-node).

CUDA OOM from Fragmentation : gpu-memory-utilization too high (0.98); PyTorch temporary allocations exceed margin. Fix: lower to 0.90-0.92, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

Security & Governance

Credential isolation via K8s Secrets, read-only model mounts, SHA256 weight verification. API gateway (Envoy/Kong/APISIX) for auth, rate limiting, max_tokens caps, audit logs. NetworkPolicy restricts access to gateway namespace. Log redaction for PII. Multi-tenant isolation via session affinity or dedicated instances.

Version Locking & Change Management

versions.lock records host driver, CUDA, image digest, PyTorch, Triton, model revision. Blue/green deployments with pre-warm, canary cutover, graceful termination (180s). Parameter changes via ConfigMap/Helm, never kubectl edit; impact assessment table, 30-min observation window with preemption/queue/TTFT regression checks.

GPU-Specific Adaptation

A100/A800: no native FP8 KV; A800 NVLink bandwidth lower → consider FP8 weight + TP=2. L40S: 48GB, no NVLink, PCIe TP scaling ~1.3-1.5x. H100/H200: FP8 KV max benefit; H200 141GB HBM allows 2x+ concurrency. Blackwell: strict version matrix (driver, CUDA, PyTorch, FlashInfer).

Cost & Capacity Planning

Cost breakdown: GPU depreciation ~70%, idle redundancy, cold-start waste, low utilization. Quantization table: BF16 baseline, FP8 weight 0.5x VRAM, FP8 weight+KV 0.5x each, 4-bit 0.25x weight. Capacity example: 200k daily req, peak QPS 8, P50 1.2K in/300 out → peak concurrency ~100 → 2-3 replicas with safety factor.

SLA Definition & Parameter Derivation

Three layers: Gateway (queue P95<500ms, 5xx<0.1%), Engine (TTFT P95<1.5s, ITL P99<80ms), Experience (end-to-end <15s). Derive ITL budget from business SLA: e.g., 400 chars ≈ 270 tokens, 8s total → ITL <26ms if TTFT=1s. Prioritize ITL over TTFT. Tail handling: separate long-request pool, gateway rejection, soft timeouts.</p><h2>Performance Regression Prevention</h2><p>Baseline JSON per release (image, model, args, concurrency, throughput, TTFT P95, ITL P99, preemptions). Regression thresholds: throughput -10%, TTFT +20%, ITL +15%, any preemption = block. Common causes: default changes, kernel backend switches, CUDA Graph bucket changes, dependency drift.</p><h2>Cross-Node Deployment</h2><p>Trigger: model too large for single node. Use TP=GPUs-per-node, PP=nodes. Preflight: ibstat, NCCL_DEBUG=INFO, all_reduce_perf (busbw >300GB/s on 400G IB). Env vars: NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME=ib0, NCCL_NET_GDR_LEVEL=2. Failure patterns: NCCL init hang (wrong interface), periodic hangs (GID index), half throughput (Socket not IB), single-node preemption (GPU throttling).</p><h2>Misconceptions Corrected</h2><ul><li>GPU utilization 100% ≠ saturated; watch memory utilization (HBM bandwidth) and SM occupancy.</li><li>Adding GPUs only helps if bottleneck is compute; KV capacity, network, gateway limits, storage may be true bottlenecks.</li><li>Parameter tuning cannot break physical limits: VRAM = weights+KV+workspace, bandwidth = bytes/token × throughput, compute = input² × layers × params.</li><li>Open-source models have license restrictions (commercial use, user caps, attribution).</li></ul><h2>Elastic Scaling & Memory Fragmentation</h2><p>Fixed replicas waste money; scale on queue depth (best proxy). Cold-start paradox (3-5 min vs 10-15 min peaks) solved by hot standby, cron pre-scale, startup optimization. Scale-down safety: drain connections, verify num_requests_running=0. New replica warm-up before joining endpoints. Fragmentation sources: variable activations, weight load/unload, CUDA Graph captures. Mitigation: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, limit CUDA Graph sizes, lower gpu-memory-utilization. Monitor reserved-allocated delta; schedule restart if >5% total VRAM.</p><h2>Log Troubleshooting Handbook</h2><p>Error categories: startup (P0), runtime crash (P0), performance degradation (P1), deprecation noise (P3). Startup errors: max-model-len > KV capacity (lower or quantize), libcudart version mismatch (align driver/image), HF connectivity (HF_HUB_OFFLINE=1), unsupported architecture (upgrade vLLM). Runtime crash: exit code 137 (container OOM) vs 1 (GPU OOM). Correlation with batch jobs at 3AM → rate-limit batch, lower max-num-seqs, enable expandable_segments.

HTTP Interface Operations

Timeout chain: engine (300s) < gateway (320s) < client (330s). Error codes: 400 bad request, 404 model not found, 422 validation (max-model-len), 429 rate limit, 500 engine error, 503 paused/overloaded. SSE stream integrity: verify [DONE] marker; track finish_reason=null as abnormal termination. Batch endpoints: limit batch size, semaphore, async queue for large jobs.

Supply Chain & Image Security

Pull only official images, pin by digest (sha256). Minimal custom Dockerfile: add healthcheck, non-root user (verify cache dir writable). Vulnerability policy: critical 1 week, medium monthly, low track. Model file integrity: MANIFEST.sha256 for all .safetensors/.json/.py; verify on startup.

Configuration Governance

Single source of truth: Git → Helm/Kustomize → kubectl apply. Drift detection via kubectl diff in CI/CD. Annotate every non-default parameter with rationale (e.g., max-model-len=8192 because P99 input 6K + 2K margin).

Scenario Templates & On-Call Quick Reference

Four templates: Knowledge QA (RAG), Code Completion, Long Doc Summary, Multi-turn Agent. Each with tuned args and key notes (prefix caching, session affinity). On-call 3-min triage: service alive? all or partial? sudden or gradual? recent changes? traffic shift? Cheat sheet maps symptoms to first checks and common causes. Communication template: impact, current action, business action needed.

Final Checklist

[ ] Version matrix locked, image by digest
[ ] Model revision & weight hash recorded
[ ] Startup log KV cache capacity & max concurrency captured
[ ] Probes: startup covers cold start, readiness uses /v1/models
[ ] Gateway: auth, quota, timeout, SSE, 429 semantics
[ ] Buffer: /dev/shm sufficient, NCCL interface explicit
[ ] Monitoring: preemption, queue, KV water, TTFT/ITL, prefix hit rate
[ ] Alerts: thresholds based on baseline
[ ] Load test: real length dist, SLA with margin
[ ] Change: canary path, rollback cmd, observation window
[ ] Security: NetworkPolicy, no creds on disk, log redaction
[ ] Docs: param comments, runbook, postmortem template