vLLM Production Tuning: The max_num_seqs, GPU Memory & Concurrency Triad

Background: Default Parameters Work but Fail Under Production Load

Running vllm serve /models/Qwen3-8B starts the service, but teams soon hit three problems: (1) P99 latency jumps from 800 ms to 12 s with Preemption and Waiting: 300 reqs logs; (2) GPU shows 92% memory usage yet throughput is only one-third of theoretical; (3) increasing --max-num-seqs from 256 to 1024 causes OOM or frequent preemption, reducing throughput. The root cause is that gpu_memory_utilization, max_model_len, and max_num_seqs are not independent knobs but a triad constraint — changing one reshapes the feasible region of the other two.

Scope & Environment Assumptions

Covers single-node single/multi-GPU (TP) OpenAI-compatible API deployments for online inference (chat, RAG, code completion) with TTFT/ITL SLOs, offline batch inference, and long-context (32K–128K) scenarios. Excludes multi-node (PD/EP) topologies, quantization accuracy evaluation, and speculative decoding deep dives. Environment: vLLM v0.23.x (V1 engine), CUDA 12.8, H100 80GB / A100 80GB / L40S 48GB, models Qwen3-8B (BF16), Llama-3.1-8B-Instruct, Qwen3-32B, Python 3.12, bare-metal systemd or Kubernetes.

Version Differences to Watch

--gpu-memory-utilization

default varies (0.90/0.92) — always verify via vllm serve --help. max_num_seqs default: SchedulerConfig class = 128, EngineArgs = None (engine derives, often 1024 in V1). max_num_batched_tokens default: SchedulerConfig = 2048, engine derives (commonly 8192).

V1 removed CPU swap preemption; only recompute preemption exists. --swap-space is ignored.

Metric vllm:gpu_cache_usage_perc renamed to vllm:kv_cache_usage_perc in V1.

Verification command:

vllm --version
vllm serve --help | grep -A2 -E "max-num-seqs|max-num-batched-tokens|gpu-memory-utilization"

Core Knowledge: Memory, KV Cache & Scheduling

V1 Engine Three-Layer Architecture

API Server Layer : HTTP handling, tokenization, validation, assembles EngineCoreRequest (CPU-bound, 1–2 processes).

EngineCore (Scheduler) : Single process holding Scheduler and KVCacheManager; all queuing, batching, preemption decisions happen here. max_num_seqs and max_num_batched_tokens constrain this layer.

Executor (Execution) : Holds model weights, runs forward. Single GPU uses UniProcExecutor; multi-GPU uses MultiprocExecutor (one worker per GPU).

Key: Scheduler is single-threaded; each step decides which requests run and how many tokens. max_num_seqs increases scheduler decision overhead but is negligible vs GPU compute.

PagedAttention & Blocks

KV cache split into fixed-size blocks (default block_size=16 tokens), allocated on-demand, logically contiguous but physically scattered. Benefits: high memory utilization (waste limited to last block ~8 tokens), prefix caching (shared blocks for common prefixes), preemption via block release + recompute, controllable fragmentation. Do not change block_size without a clear reason.

GPU Memory Breakdown (H100 80GB, gpu_memory_utilization=0.90 → 72 GB budget)

Component                Typical Size (8B BF16)   Notes
Model Weights            ~16 GB                   8B × 2 bytes + embedding/lm_head
Activations              1–4 GB                   Scales with max_num_batched_tokens
CUDA Graph               0.5–2 GB                 Scales with max_num_seqs and captured batch sizes
NCCL/Comm Buffers        0.2–1 GB/GPU             Only when TP > 1
Temp Workspace           0.5–2 GB                 cuBLAS/cuDNN/FlashAttention scratch
KV Cache                 Remainder                vLLM allocates all leftover memory to KV pool

Key insight: KV cache is a residual , not a reserved amount. Startup: load weights → dummy forward to measure activation peak → remaining memory becomes KV pool.

KV pool ≈ gpu_mem_util × total_mem − weights − activation_peak − graph − workspace − nccl

Therefore, increasing max_num_seqs does not increase memory usage — it only raises the scheduling ceiling. Pool size is driven by gpu_memory_utilization and max_model_len (affects activation peak and profiling reservation).

Startup Profiling Logs — The Tuning Baseline

INFO Available KV cache memory: 48.30 GiB
INFO GPU KV cache size: 350,208 tokens
INFO Maximum concurrency for 8,192 tokens per request: 42.75x

Available KV cache memory : Memory left for KV after weights, activations, graph.

GPU KV cache size : Total token capacity of the KV pool (not concurrency).

Maximum concurrency for N tokens/request : KV cache size / N. If this is only 2.3x, the GPU cannot handle long-context high concurrency — must reduce max_model_len, add GPUs, or enable FP8 KV.

Per-Token KV Memory Formula (GQA/MHA)

bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

2

: K and V each. num_layers: Transformer layers. num_kv_heads: KV heads (much smaller than Q heads in GQA). head_dim: Dimension per head. bytes_per_element: 2 for BF16/FP16, 1 for FP8.

Real-model calculations:

Model           Layers   KV Heads   Head Dim   BF16/token   FP8/token
Llama-3.1-8B    32       8          128        128 KiB      64 KiB
Qwen2.5-7B      28       4          128        56 KiB       28 KiB
Qwen3-8B        36       8          128        144 KiB      72 KiB
Qwen3-32B       64       8          128        256 KiB      128 KiB

Example Qwen3-8B: 2 × 36 × 8 × 128 × 2 = 147,456 bytes = 144 KiB/token. 48.30 GiB pool → ~351,600 tokens (log shows 350,208; difference due to block alignment and metadata). Formula matches order-of-magnitude; trust log for exact value.

Two Types of Concurrency

HTTP concurrent connections : API server level (uvicorn workers, queue) — rarely the bottleneck.

Scheduling concurrency ( max_num_seqs ) : Hard limit of sequences per scheduler step; excess requests queue.

Memory-bound concurrency : KV cache size / avg tokens per request. Physical limit; exceeding triggers preemption.

Actual sustainable concurrency = min(scheduling_concurrency, memory_concurrency). Goal: bring both close without either becoming a bottleneck.

Example: Qwen3-8B, H100, KV pool 350K tokens, avg request 2000 tokens (1500 in + 500 out). Memory concurrency ≈ 175. If max_num_seqs=128, scheduling caps at 128 (memory underused). If max_num_seqs=512, memory saturates → frequent preemption. Practical max_num_seqs ≈ 1.2–1.5 × memory concurrency (since KV grows gradually), but must calibrate via preemption metrics.

Parameter Coupling Constraints

gpu_memory_utilization ↑ → KV pool ↑ → memory concurrency ↑
max_model_len ↑ → per-request KV ↑ → memory concurrency ↓
max_num_seqs ↑ → scheduling concurrency ↑ (but capped by memory concurrency)

Hard constraints: max_num_seqs ≤ max_num_batched_tokens (validated at startup). max_num_seqs × avg_tokens_per_request ≤ KV cache size (else preemption). max_model_len ≤ model_max_len with output headroom. gpu_memory_utilization not too high (leave headroom for driver, monitoring, other processes). Production: 0.85–0.92; 0.98 risks allocation failures.

Preemption in V1: Recompute Only

When KV pool exhausted, scheduler releases a running request's KV blocks; request restarts prefill later. Cost: generated tokens discarded (output kept), TTFT/TPOT spikes, GPU does redundant work → effective throughput drops. Logs show WARNING ... Preemption occurred; metric vllm:num_preemptions_total increases. Diagnosis: kv_cache_usage_perc > 0.95 sustained + preemption → reduce max_num_seqs or max_model_len, add GPUs, or enable FP8 KV. Do not raise gpu_memory_utilization to brute-force.

Chunked Prefill: TTFT vs ITL Trade-off

V1 enables chunked prefill by default: long prompt prefill split into chunks, interleaved with decode requests in same batch. Controlled by max_num_batched_tokens (max tokens per step). Scheduler fills decode requests (1 token each) first, remaining budget goes to prefill.

Setting          TTFT          ITL          Throughput
Small (e.g., 2048)       Poor (prefill fragmented)   Good (decode less interference)   Low
Large (e.g., 16384)      Good (prefill in fewer steps)   Poor (decode starved)   High
Very large (≥32768)      Best        Worst        Highest, but latency unpredictable

Heuristics: Online chat (TTFT < 2s, ITL < 100ms) → 8192; Long doc summary (TTFT insensitive) → 16384–32768; Code completion (ITL critical) → 2048–4096. If disabling chunked prefill ( --no-enable-chunked-prefill), max_num_batched_tokens must exceed max_model_len or startup crashes.

Prefix Caching: Free Lunch for RAG

Enabled by default in V1 (disable with --no-enable-prefix-caching). Identical prefixes share KV blocks. RAG (fixed system prompt + long doc prefix) sees 60–90% hit rate. No benefit for random requests. Overhead: block ref-count management; cached-but-unused blocks still consume KV pool, potentially worsening memory pressure. Monitor hit rate:

rate(vllm:prefix_cache_hits[5m]) / rate(vllm:prefix_cache_queries[5m])

< 10% → consider disabling; > 50% → keep enabled.

End-to-End Tuning Workflow

1. Baseline → 2. Theoretical Ceiling → 3. Load Test → 4. Find Bottleneck → 5. Change One Param → 6. Regress → 7. Freeze Config

Baseline : Current QPS, TTFT P50/P99, ITL P50/P99, kv_cache_usage_perc peak, preemption count.

Theoretical ceiling : Compute KV pool & memory concurrency via formulas; compare with baseline to distinguish parameter misconfig vs hardware limit.

Load test : Use vllm bench serve with target workload profile (not random saturation).

Bottleneck analysis : Correlate num_requests_waiting, kv_cache_usage_perc, preemptions, GPU utilization.

Change one parameter : Single-variable changes for attribution.

Regression : Re-run identical load test, compare three key metrics.

Freeze : Write into systemd/Helm/config-center with rationale comments.

Production Deployment: Bare-Metal to Kubernetes

Pre-flight Checks

# Driver & GPU
nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv
# Residual processes
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
# Container runtime
nvidia-ctk --version 2>/dev/null || echo "no nvidia-ctk"
docker info 2>/dev/null | grep -i runtime
# Shared memory (critical for TP>1)
df -h /dev/shm
# Hugepages & file descriptors
cat /proc/meminfo | grep -i huge
ulimit -n

Expectations: GPU count/model matches, memory.total 81559 MiB (H100) or 81920 (A100), no leftover processes, /dev/shm ≥ 16 GB (Docker default 64 MB breaks TP), ulimit -n ≥ 65535.

Annotated Production Startup Command (Qwen3-8B, H100, Online Chat)

vllm serve /models/Qwen3-8B \
  --served-model-name qwen3-8b \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --api-key "${VLLM_API_KEY}" \
  --disable-log-stats

Rationale per parameter: --max-model-len 32768: Model supports 128K, but set to 32K. max_model_len directly caps per-request KV and affects profiling activation peak. 128K would quarter max concurrency; most requests never use it. Set to business P99 context × 1.5, not model max. --gpu-memory-utilization 0.90: Reserve 10% for driver, monitoring agents, CUDA context fluctuations. Lower to 0.80 if co-locating other processes. >0.95 risks fragmentation-induced allocation failures over long runs. --max-num-seqs 256: KV pool ~350K tokens, avg request 2000 tokens → memory concurrency ~175. Since KV grows gradually, instantaneous usage < terminal value, so set ~1.4× (256). Accept minor preemption at peak; monitor via metrics. --max-num-batched-tokens 8192: Balances TTFT/ITL for chat. Lower to 4096 if ITL P99 violates SLO; raise to 16384 if TTFT high and ITL has headroom. --enable-prefix-caching: Essential for RAG/multi-turn; disable for purely random workloads. --api-key from env var (not hardcoded) to avoid leakage in ps and shell history. --disable-log-stats: Suppress per-step stats I/O. Enable during tuning (remove flag) to observe throughput/queueing.

Multi-GPU (TP=4) differences: max_num_seqs is global (not per-GPU); each rank holds full KV shard so pool sums across GPUs. All 4 ranks must start simultaneously. /dev/shm must be large enough or NCCL fails with RuntimeError: NCCL error or Bus error.

systemd Unit

[Unit]
Description=vLLM Qwen3-8B OpenAI API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm
EnvironmentFile=/etc/vllm/qwen3-8b.env
ExecStart=/opt/vllm/venv/bin/vllm serve /models/Qwen3-8B \
  --served-model-name qwen3-8b \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching
TimeoutStartSec=900
TimeoutStopSec=60
Restart=on-failure
RestartSec=10
KillSignal=SIGINT
KillMode=mixed
LimitNOFILE=65535
LimitMEMLOCK=infinity
StandardOutput=journal
StandardError=journal
SyslogIdentifier=vllm-qwen3-8b

[Install]
WantedBy=multi-user.target

Env file ( chmod 600, chown vllm:vllm):

VLLM_API_KEY=sk-xxxxxxxxxxxxxxxx
HF_HOME=/data/hf-cache
VLLM_LOGGING_LEVEL=INFO

Design notes: KillSignal=SIGINT + KillMode=mixed for graceful shutdown (vLLM handles SIGINT better than SIGTERM). TimeoutStartSec=900 accommodates 3–10 min model load. LimitMEMLOCK=infinity required for NCCL page-locked memory in TP.

Kubernetes Deployment

Key points: startupProbe with failureThreshold: 90 + periodSeconds: 10 = 15 min startup window (model load + profiling + CUDA graph capture). Without it, liveness probe kills starting pods causing restart loops. emptyDir: medium: Memory for /dev/shm (16 GiB) — default 64 MB breaks NCCL. maxUnavailable: 0 ensures rolling update starts new pod before stopping old (requires double GPU capacity). nodeSelector on nvidia.com/gpu.product (from GPU Feature Discovery) for precise GPU type targeting.

Reading Startup Logs

INFO Starting vLLM API server on http://0.0.0.0:8000
INFO Loading model weights took 14.82 GB
INFO Available KV cache memory: 48.30 GiB
INFO GPU KV cache size: 350,208 tokens
INFO Maximum concurrency for 32,768 tokens per request: 10.69x
INFO init engine (profile, create kv cache, warmup model) took 16.5 s

Loading model weights : Actual weight footprint (differs from theoretical due to shared embeddings, dtype variations).

Available KV cache memory : Most critical number. H100 80GB total 79.65 GiB × 0.90 = 71.69 GiB budget; minus 14.82 GiB weights, ~8.5 GiB activations/graph/workspace → 48.30 GiB.

GPU KV cache size : Token capacity of KV pool. Cross-check: 350,208 × 147,456 bytes ≈ 51.6 GB ≈ 48.1 GiB (matches within block alignment loss). This line is the sole basis for judging target concurrency feasibility.

Maximum concurrency for 32,768 tokens : 350,208 / 32,768 = 10.69. If business P99 context is 4K, actual concurrency ≈ 350,208 / 4096 ≈ 85.

Init engine time : Includes profiling & CUDA graph capture. Larger max_num_seqs → more capture batch sizes → longer (256 ~16s, 1024 >60s). K8s startupProbe must accommodate.

Benchmarking with vllm bench serve

Do not use ab or wrk — they don't understand streaming. Use built-in benchmark:

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model qwen3-8b \
  --dataset-name random \
  --random-input-len 1500 --random-output-len 500 \
  --num-prompts 500 --max-concurrency 128 --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --save-result --result-dir ./bench-results

Key flags: --dataset-name random (synthetic, controllable lengths; use sharegpt or custom for realism). --random-input-len/--random-output-len must match real distribution — wrong lengths invalidate entire tuning . --num-prompts ≥ 10× concurrency to smooth cold-start noise. --max-concurrency set above expected to make server the bottleneck. --request-rate default inf (unthrottled); use fixed rate (e.g., 20) to simulate real QPS. --ignore-eos forces generation to exact output length, avoiding early-stop bias.

Output interpretation:

Output token throughput (tok/s) : Primary throughput metric. H100 8B BF16 typically 1500–3000 tok/s depending on I/O ratio.

P99 TTFT : Queueing + prefill. P99 > 5× P50 indicates queueing or long-prompt blocking.

P99 TPOT/ITL : Decode stability. P99 > 3× P50 usually means preemption or batch variance.

Request throughput (QPS) : View with token throughput to distinguish "many short" vs "few long" requests.

Pitfalls: Client & server on same machine contend CPU — use separate load generator. Too few prompts inflates P99 via cold CUDA graph runs. Missing --ignore-eos yields artificially high throughput from early stops.

Metrics Collection (Prometheus)

vLLM exposes /metrics. Core metrics:

Metric                              Type    Meaning                          Use
vllm:num_requests_running         Gauge   Active requests                  Actual concurrency
vllm:num_requests_waiting         Gauge   Queued requests                  Overload detection
vllm:kv_cache_usage_perc          Gauge   KV cache usage (0–1)             Memory pressure
vllm:prefix_cache_queries         Counter Prefix cache lookups             Hit rate calc
vllm:prefix_cache_hits            Counter Prefix cache hits                Hit rate calc
vllm:time_to_first_token_seconds  Histogram TTFT                           SLO
vllm:inter_token_latency_seconds  Histogram ITL                            SLO
vllm:request_prompt_tokens        Histogram Input length distribution        Capacity calibration
vllm:request_generation_tokens    Histogram Output length distribution       Capacity calibration
vllm:prompt_tokens_total          Counter Cumulative input tokens          Throughput stats
vllm:generation_tokens_total      Counter Cumulative output tokens         Throughput stats
vllm:request_success_total        Counter Successes by finished_reason (stop/length/abort) Success rate; abort ratio = client cancels/timeouts

Notes: V1 uses kv_cache_usage_perc (old gpu_cache_usage_perc deprecated). Preemption metric name varies — trust /metrics output. /metrics typically unauthenticated even with --api-key; don't expose publicly if behind gateway.

Five Real-World Tuning Cases

Case A: Startup OOM — KV Cache Cannot Be Allocated

Symptom :

ValueError: ... 8.42 GiB KV cache needed > available 6.13 GiB

or torch.OutOfMemoryError.

Root causes : (1) Other processes holding GPU memory; (2) max_model_len too large — single-request KV exceeds available pool; (3) gpu_memory_utilization too high — profiling exhausts memory.

Diagnosis : Check nvidia-smi --query-compute-apps; inspect logs for Available KV cache memory and model weights; test minimal config (

--max-model-len 4096 --gpu-memory-utilization 0.85 --max-num-seqs 8

).

Fixes : If available KV < 2 GiB → lower max_model_len or enable --kv-cache-dtype fp8 (halves KV). If available OK but OOM → kill stray processes. If only works at 4096 → GPU insufficient for target context; consider FP8 KV, TP split, or larger GPU.

FP8 KV usage: --kv-cache-dtype fp8 --calculate-kv-scales (calibrates scales at startup, slight startup penalty, doubles KV capacity). Verify via log: GPU KV cache size should double. Rollback: remove flag, restart.

Case B: GPU Util 95% but Throughput Half of Expected

Symptom : nvidia-smi shows 95%+ GPU util; num_requests_running ~40 vs max_num_seqs=256; num_requests_waiting ~0; output throughput only 800 tok/s.

Root cause : running low, waiting zero → client not driving enough load. Tuning max_num_seqs useless.

Diagnosis : Check client --max-concurrency; server metrics num_requests_running/waiting, kv_cache_usage; request length histograms.

Fixes : Increase client concurrency or upstream QPS. If running capped at a value with growing waiting → raise max_num_seqs per memory concurrency estimate. If running high but tokens/step low → raise max_num_batched_tokens to 8192–16384. Short requests (<50 output tokens) naturally yield high GPU util but low throughput (launch overhead). Small models (<8B) cannot saturate H100 — consider larger model, multi-instance (MIG/MPS), or speculative decoding.

Case C: High TTFT, P99 at 8 Seconds

Symptom : P50 TTFT 400 ms, P99 8 s; num_requests_waiting spikes; occasional Preemption logs.

Root cause : TTFT = queue time + prefill time. P99 ≫ P50 → queueing, not slow prefill.

Diagnosis : Check kv_cache_usage_perc near 1.0 + preemption → memory pressure; kv_cache_usage_perc < 0.8 + high waiting → max_num_seqs bottleneck; input length histogram long-tail (P99 20× P50) → few long requests block queue.

Fixes : Memory pressure → lower max_num_seqs, enable FP8 KV, or reduce max_model_len. Scheduling limit → raise max_num_seqs (if memory allows). Long-tail inputs → client-side length limit or dedicated long-context pool. Small max_num_batched_tokens → raise to 16384. Chunked prefill interference (ITL ok, TTFT high) → increase prefill budget.

Long-request isolation: Pool A ( max_model_len=8192, max_num_seqs=256) for 90% traffic; Pool B ( max_model_len=131072, max_num_seqs=8) for long docs; gateway routes by prompt_tokens. Trade-off: Pool B often idle → lower resource efficiency.

Case D: ITL Jitter, P99 5× P50

Symptom : TPOT P50 28 ms, P99 140 ms; users report "typewriter stutter"; kv_cache_usage_perc oscillates 0.85–0.99.

Root causes : (1) Preemption (request kicked, recompute gap); (2) Batch size swings (e.g., 200 → 20); (3) Long prefill chunks starving decode.

Diagnosis : Log preemption count; watch kv_cache_usage_perc volatility; check batch size via Running: N reqs logs.

Fixes : Preemption + high KV → lower max_num_seqs to memory concurrency minus 20% margin. Prefill interference → lower max_num_batched_tokens to 4096. Bimodal request lengths → length-based pools. CUDA graph recapture on batch changes → keep max_num_seqs stable.

Trade-off : Reducing max_num_seqs from 256 to 160 eliminates preemption, stabilizes ITL, but may drop throughput 10–20%. Hard SLO (voice, code completion) → prioritize stability; offline batch → prioritize throughput.

Case E: Long Context Kills Concurrency

Symptom : --max-model-len 131072; log shows Maximum concurrency for 131,072 tokens per request: 2.67x; two long requests saturate memory, third queues indefinitely, existing requests preempt.

Root cause : Setting max_model_len to model max inflates per-request KV ceiling, collapsing memory concurrency to single digits.

Hidden costs : (1) Profiling uses near- max_model_len dummy input → reserves extra activation memory, shrinking KV pool. (2) Scheduler may conservatively pre-allocate for worst-case.

Fixes : Set max_model_len to business P99 × 1.5 (e.g., P99=8K → 16384). Offload ultra-long requests to dedicated instance. Enable prefix caching to reuse common long prefixes.

Verification : Log Maximum concurrency should jump from 2.67x to >20x (at 16384). Monitor rejected errors (vLLM returns 400 for > max_model_len).

Memory Waterline Operations

Waterline Tiers

Waterline   Meaning   Action
< 0.60      Idle      Raise max_num_seqs or co-locate services
0.60–0.85   Healthy   Maintain
0.85–0.95   Tight     Watch preemption & ITL; plan scale-out
> 0.95      Danger    Preemption inevitable; immediately lower concurrency or scale

High waterline ≠ problem if SLO met and no preemption. 0.93 can be healthy (high utilization). Combine waterline with preemption & latency for decisions.

Why Not gpu_memory_utilization=0.98

CUDA context creation needs hundreds of MiB; 0.98 leaves zero margin for new processes.

Long-run fragmentation causes large-block allocation failures despite free total.

Monitoring agents, nvidia-smi, log collectors sporadically allocate GPU memory.

Production range 0.85–0.92. 0.94 acceptable only if occasional OOM/restart tolerated.

Co-location Risks

Running two vLLM instances on one GPU with gpu_memory_utilization=0.45 each (sum 0.9) risks: simultaneous activation peaks exceeding physical memory; mutual KV pressure causing cross-instance OOM; single GPU failure (XID, OOM) takes down both. Safer: MIG hard partitioning or separate GPUs.

Alerting Rules (Prometheus)

- alert: VLLMKVCacheHigh
  expr: vllm:kv_cache_usage_perc > 0.95
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "vLLM KV cache waterline {{ $value | humanizePercentage }}"
    description: "Sustained high waterline triggers preemption; check if max_num_seqs exceeds memory concurrency"

- alert: VLLMQueueBacklog
  expr: vllm:num_requests_waiting > 50
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "vLLM queue backlog {{ $value }} requests"
    description: "Combine with running to distinguish concurrency limit vs memory pressure"

- alert: VLLMTTFTHigh
  expr: |
    histogram_quantile(0.99,
      sum by (le, instance) (rate(vllm:time_to_first_token_seconds_bucket[5m]))
    ) > 5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "TTFT P99 exceeds 5 seconds"

Thresholds must reflect business SLO (5s unacceptable for chatbot, fine for batch).

Capacity Planning: From Concurrency to GPU Count

Three Required Inputs

Peak QPS (q) : Requests/second at peak (not daily average).

Avg input/output lengths (L_in, L_out) : From production logs (P50 and P90).

SLO : TTFT and TPOT upper bounds.

Using Little's Law: Concurrency = QPS × Avg Latency, where Avg Latency ≈ TTFT + L_out × TPOT.

Example: QPS=20, TTFT=0.5s, L_out=500, TPOT=30ms → Latency = 0.5 + 500×0.03 = 15.5s → Concurrency = 310.

Single-GPU Concurrency Capacity

Single-GPU concurrency = KV cache tokens / (L_in + L_out)

Qwen3-8B + H100: 350,208 tokens / 2000 = 175 concurrent requests.

Required GPUs = ceil(310 / 175) = 2, plus 30% headroom → 3 GPUs.

Reference Capacity Table (H100 80GB, BF16, 0.90 utilization)

Model           KV/token   KV Pool   2K Context Concurrency   8K Context Concurrency   32K Context Concurrency
Llama-3.1-8B    128 KiB    ~390K     195                      48                       12
Qwen2.5-7B      56 KiB     ~880K     440                      110                      27
Qwen3-8B        144 KiB    ~350K     175                      43                       10
Qwen3-32B (TP=4) 256 KiB   ~750K (4 GPUs) 375                  94                       23

Insights: (1) KV structure dictates capacity — Qwen2.5-7B's 4 KV heads yield 2.5× concurrency of Qwen3-8B on same hardware. Model selection must weigh KV size alongside quality. (2) Context length is inverse to concurrency — 2K→32K drops concurrency 16×. "Supports 128K" ≠ "supports 128K at high concurrency".

Throughput-Driven Sizing

Concurrency alone insufficient; must satisfy token throughput:

Required token throughput = QPS × (L_in + L_out)
Single-GPU token throughput = measured (output + input tok/s)

Example: QPS=20, L_in=1500, L_out=500 → 40,000 tok/s needed. H100 Qwen3-8B measured ~5,000–8,000 tok/s → 40,000 / 6,000 ≈ 7 GPUs. This often exceeds concurrency-based estimate (3 GPUs). Choose max(concurrency_GPUs, throughput_GPUs) × 1.3 safety factor. Many teams scale only for concurrency and still queue because throughput is the real bottleneck.

Validation via Sustained Load Test

vllm bench serve --backend vllm --base-url http://127.0.0.1:8000 \
  --model qwen3-8b --dataset-name random \
  --random-input-len 1500 --random-output-len 500 \
  --num-prompts 36000 --request-rate 20 --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el

Run 30 min at target QPS. Watch: P99 TTFT/ITL within SLO; num_requests_waiting stable (not growing); kv_cache_usage_perc stable 0.7–0.9; zero preemptions. Growing waiting → capacity shortfall. Zero waiting but high latency → per-request slowness (check max_num_batched_tokens or model).

Risk, Rollback & Canary

Parameter Change Risk Levels

Parameter                     Impact              Risk                Restart Required
gpu_memory_utilization        KV pool size        Medium (OOM if too high)   Yes
max_model_len                 Per-request KV cap, concurrency  Medium (lower rejects long requests)  Yes
max_num_seqs                  Scheduling concurrency  Low–Medium (high causes preemption)  Yes
max_num_batched_tokens        TTFT/ITL trade-off  Low                 Yes
enable_prefix_caching         Hit rate, memory    Low                 Yes
kv_cache_dtype                KV capacity, accuracy  Medium (accuracy evaluation needed)  Yes
tensor_parallel_size          Topology            High (deployment change)  Yes

All changes require process restart (tens of seconds to minutes). Must use canary.

Canary Strategies

K8s RollingUpdate (preferred) : maxSurge: 1, maxUnavailable: 0 — new pod ready before old terminated. Requires spare GPU capacity; else pods stick in Pending.

Dual Deployment + Gateway Weight Shift : Old v1 (3 replicas), new v2 (1 replica). Gateway (nginx) weights 9:1 → 5:5 → 0:10 over 30-min observation windows. Instant rollback by weight change.

Bare-metal systemd : Validate in staging; production: stop one instance, update, start, verify; repeat. Capacity dips during window — schedule in low-traffic period.

Rollback

Config rollback = restore previous config file + restart. Always backup before change:

sudo cp -a /etc/vllm/qwen3-8b.env /etc/vllm/qwen3-8b.env.$(date +%Y%m%d%H%M)

K8s: kubectl rollout undo deploy/vllm-qwen3-8b (depends on revisionHistoryLimit, default 10).

Change Governance

Window: low-traffic, avoid major events/releases.

Two-person review (executor + verifier); config in Git.

Record: what changed, why, benchmark data, rollback command.

Post-change: monitor core metrics every 5 min for 30 min.

Verification Checklist

# 1. Service health
curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer ${VLLM_API_KEY}"

# 2. KV capacity & concurrency ceiling (startup logs)
journalctl -u vllm-qwen3-8b | grep -E "Available KV cache|GPU KV cache size|Maximum concurrency"

# 3. Effective config matches intent
ps -ef | grep "vllm serve" | grep -v grep

# 4. Key metrics
curl -s http://127.0.0.1:8000/metrics | grep -E "^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc)"

# 5. Benchmark baseline (re-run after every change, archive history)
vllm bench serve --backend vllm --base-url http://127.0.0.1:8000 \
  --model qwen3-8b --dataset-name random \
  --random-input-len 1500 --random-output-len 500 \
  --num-prompts 500 --max-concurrency 128 --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el --save-result

# 6. Error rate
curl -s http://127.0.0.1:8000/metrics | grep request_success_total

# 7. GPU health (no throttling/ECC)
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,temperature.gpu,clocks.sm,clocks.max.sm --format=csv

Pass criteria: /health 200; /v1/models lists served-model-name; startup KV capacity matches expectation; P99 TTFT/ITL within SLO; abort ratio < 1%; GPU no throttling/ECC errors.

Common Misconceptions

max_num_seqs bigger is better — Only raises scheduling ceiling; exceeding memory concurrency causes preemption, lowering throughput. Set via formula with 10–20% headroom.

gpu_memory_utilization=0.98 yields many more requests — Marginal KV gain (<5%) but zero headroom for context creation, fragmentation, agent allocations. Risk outweighs reward.

max_model_len = model max is safest — Inflates per-request KV reservation and profiling peak, slashing actual concurrency. Set to business P99 × 1.5.

max_num_batched_tokens bigger = higher throughput — Throughput rises but ITL degrades. For online services, this trades user experience for throughput. Choose by SLO, not max value.

Prefix caching always helps — Only when prefix repetition high. Random workloads: near-zero hit rate, management overhead, cached blocks consume KV pool → lower effective concurrency. Watch hit rate metric.

Benchmark numbers = production capacity — Benchmarks use fixed lengths, no network jitter, no upstream/downstream dependencies. Production P99 typically 30–50% worse. Plan with headroom; observe two weeks post-launch before finalizing.

Summary

vLLM tuning is not parameter memorization but understanding the constraint chain:

Memory budget → weights/activations/graph overhead → KV pool → per-request KV → memory concurrency → actual concurrency

Every link is calculable and measurable. Five actions to remember:

Read the three startup log lines ( Available KV cache, GPU KV cache size, Maximum concurrency) — they are the ground truth.

Estimate with formula, calibrate with logs : 2 × layers × kv_heads × head_dim × dtype_bytes per token; match log → mental model validated.

Set max_num_seqs by memory concurrency, not intuition : slightly above KV tokens / avg request length, with 10–20% buffer.

Set max_model_len by business P99, not model max — the most wasted parameter.

Change one parameter at a time, always benchmark regression — otherwise no attribution, no safe rollback.

Final reminder: Tuning solves only half of performance problems. The other half lies in model choice (KV structure), hardware (memory bandwidth), deployment topology (TP/PD separation), and request distribution (long/short isolation). When parameters are maxed out and SLOs still missed, it's time to change architecture — not keep turning screws.