LLM Quantization in Production: Choosing Between GPTQ, AWQ, and GGUF with Benchmarks

Part 1: Problem Background — Memory Pressure Isn't Solved by Just Switching Quantization Formats

The author recounts a real incident: a team running Qwen2.5-72B-Instruct on 8×A100 80GB with AWQ 4-bit via vLLM faced three issues after scaling to 4 replicas: (1) TTFT P99 spiked from 800ms to 6s while vllm:num_requests_running hit concurrency limits; (2) replicas OOM-killed by host kubelet 10-20 minutes after peak; (3) quantized model quality "not terrible but worse than expected" — long-text summaries missed key info.

Root causes: (1) TTFT explosion was prefill queuing due to KV cache blocks exhausted by long-context requests ( vllm:kv_cache_usage_perc at 0.99), not weight memory; (2) OOM was host-level OOM killer because vLLM's startup peak (weight load + format conversion + KV cache allocation) exceeded container memory limit under --gpu-memory-utilization 0.95 on shared nodes; (3) quality degradation was never quantified — no perplexity, task accuracy, or long-context recall baselines.

Key insight: quantization is an engineering decision chain covering format selection, precision evaluation, memory planning, kernel choice, and serving parameter tuning — not just "16-bit to 4-bit".

Part 2: Version & Environment Boundaries — Pin Versions Before Drawing Conclusions

Quantization ecosystem is fragile; minor version changes break compatibility. All conclusions based on:

Component          Version      Release Date   Notes
vLLM               0.33.0       2026-11-02   CUDA 13.0 wheels only; Python 3.11-3.14
llm-compressor     0.15.0       2026-10-28   Official quantization entry, replaces AutoAWQ
GPTQModel          1.12.x       2026         ModelCloud maintained, AutoGPTQ successor
llama.cpp          b12317       2026-11-12   GGUF reference implementation
compressed-tensors bundled with llm-compressor   vLLM's quantization format layer
PyTorch            2.14.0       2026-10      Bound to vLLM 0.33.0
CUDA               13.0         —            vLLM 0.33.0 prebuilt only for cu13

Critical boundaries:

AutoAWQ deprecated — vLLM docs mark it deprecated; functionality moved to llm-compressor. Old checkpoints ( quant_method: "awq") still load for compatibility.

AutoGPTQ unmaintained — replaced by GPTQModel (package gptqmodel, class GPTQModel, config GPTQConfig). Old imports ( AutoGPTQForCausalLM) won't work.

vLLM 0.33.0 only provides CUDA 13 wheels — if driver is 12.x, must upgrade driver, compile from source, or stay on older vLLM.

GGUF in vLLM is experimental — official docs: "highly experimental and under-optimized"; moved to OOT plugin vllm-gguf-plugin. Suitable for memory-constrained fallback, not production primary.

Part 3: Core Knowledge — What Quantization Actually Does

3.1 Mathematical Essence

Mapping FP16 weight matrix W (2 bytes/element) to INT4 (0.5 bytes) via affine transform:

W_q = round(W / s) + z
W_approx = (W_q - z) * s

s

= scale, z = zero point (zero for symmetric). Group-wise quantization: each row split into groups of 128 elements sharing one scale. Average bits per element = 4 + 16/128 = 4.125 bits → theoretical compression ~3.88×. Smaller group_size → better precision, worse compression.

3.2 GPTQ: Hessian-Based Error Compensation

Layer-wise, column-wise quantization; error from quantized column distributed to remaining columns via inverse Hessian. Requires calibration data (e.g., c4, ultrachat). 8B model ~20-40 min on single A100; 70B several hours on 8 GPUs. Without Marlin kernel, GPTQ inference can be slower than FP16 due to dequant overhead.

3.3 AWQ: Protecting 1% Important Weights

Observes few channels (0.1%-1%) disproportionately affect output. Scales important channels up before quantization, then scales activations down equivalently. Scale search via grid search on activation stats — no backprop, so 2-3× faster than GPTQ. In vLLM, AWQ kernels (GEMM/GEMV, Marlin) more mature; often higher throughput than GPTQ+Marlin, but gap varies by version.

3.4 GGUF: A Format, Not an Algorithm

GGUF (GPT-Generated Unified Format) is llama.cpp's container format — self-contained file with weights, tokenizer, hyperparameters, template. Supports multiple quantization types named Q<bits>_<variant>:

Q2_K ~2.6 bits — Ultra-low, "just runs"

Q3_K_M ~3.9 bits — Low, long-context fragile

Q4_K_M ~4.8 bits — Most common balance; K-quant mixed precision + important layers 5-bit

Q5_K_M ~5.7 bits — Near-FP16 precision, ~35% size

Q6_K ~6.6 bits — Near-lossless on most tasks

Q8_0 ~8.5 bits — ~FP16 equivalent, ~53% size

Note: Q4_K_M averages 4.8 bits, not 4 — mixed precision. Comparing "GGUF Q4" to "AWQ 4-bit" is apples-to-oranges; Q4_K_M often outperforms AWQ 4-bit at similar size. GGUF strengths: broad deployment (CPU, ARM, Apple Metal, CUDA, ROCm, Vulkan, SYCL). Weakness: GPU throughput far below vLLM due to lack of PagedAttention continuous batching.

3.5 Positioning Matrix

Type : GPTQ (Algorithm), AWQ (Algorithm), GGUF (Format holding many algos)

Calibration data : GPTQ Yes, AWQ Yes, GGUF No (weight-only)

Quant time (8B) : GPTQ 30-60 min, AWQ 15-30 min, GGUF 2-30 min

Maintenance : GPTQModel active, llm-compressor active, llama.cpp very active

vLLM support : GPTQ Full, AWQ Full (recommended), GGUF Plugin experimental

Key kernels : GPTQ Marlin/ExLlama, AWQ Marlin/GEMM, GGUF own implementations

Typical use : GPTQ Legacy/ExLlama, AWQ vLLM production, GGUF Edge/CPU/single-node

Precision (4-bit) : GPTQ Medium, AWQ Medium-high, GGUF Q4_K_M medium-high, Q2_K low

One-liner: GPU high-throughput → AWQ (vLLM) + Marlin; legacy checkpoints/ExLlama → GPTQ; CPU/edge/single-node → GGUF.

Part 4: Overall Approach — Treat Quantization as a Verified Change

Seven phases with deliverables and acceptance criteria:

1. Goal Definition   → Memory/throughput/quality targets (quantifiable)
2. Baseline Measure  → FP16 memory/throughput/quality numbers
3. Candidate Gen     → 2-3 quantization schemes, produce checkpoints
4. Quality Eval      → Perplexity + task metrics + long-context + edge cases
5. Perf Benchmark    → Fixed-load TTFT/TPOT/throughput/memory peak
6. Canary Deploy     → Small traffic shadow, dual-run
7. Rollback Plan     → One-command revert to FP16 or previous checkpoint

Order cannot be shuffled; skipping 1&2 is most common mistake.

4.1 Goal Definition: Must Be Metric-Driven

Memory: single replica weight ≤ 40 GiB, support 8K context × 64 concurrency
Throughput: fixed 32 concurrency, 512 in / 256 out → output tok/s ≥ 1200
Latency: TTFT P95 ≤ 1.2s, TPOT P95 ≤ 60ms
Quality: MMLU 5-shot drop ≤ 1.0 pp; custom biz eval drop ≤ 2.0 pp; perplexity (wikitext-2) rise ≤ 5%

4.2 Baseline Measurement: Three Sets of Numbers

Run FP16 baseline first, capture:

Memory baseline : vllm serve with --dtype bfloat16, note startup log lines: model weights take XX GiB, KV cache size: XX tokens, maximum concurrency for 8192 tokens per request: XX.XXx.

Throughput baseline :

vllm bench serve --dataset-name random --random-input-len 512 --random-output-len 256 --num-prompts 200 --max-concurrency 32

; record request/output throughput, mean/P99 TTFT, TPOT.

Quality baseline :

lm_eval --model vllm --tasks wikitext,mmlu --num_fewshot 5 --batch_size auto

; must add add_bos_token=true for quantized models; lm-eval[api]>=0.4.12.

Part 5: Practice 1 — AWQ Quantization (llm-compressor Path)

5.1 Environment Isolation: Two Separate venvs

Official docs require separate envs for llm-compressor and vllm due to conflicting torch/transformers pins.

5.2 Calibration Data: Upper Bound of Quantization Quality

Principle: calibration distribution must match production input distribution. Guidelines:

General chat → HuggingFaceH4/ultrachat_200k Chinese biz → sampled from production logs (desensitized) or shibing624/medical, BAAI/Infinity-Instruct Code models → bigcode/the-stack sample

Fine-tuned models → prefer fine-tuning training data sample

Start with 512 samples, seq len 2048; increase to 1024-2048 if quality drops.

5.3 AWQ Quantization Script

Two-step recipe: AWQModifier (smooth + scale search) then QuantizationModifier (actual quantization). Key parameters: scheme (W4A16, W4A16_ASYM, W8A8, FP8_W8A8), group_size (default 128), ignore=["lm_head"]. Save quant_meta.json for traceability.

5.4 Artifact Verification

File size: 7B 4-bit + g128 ~4.3-4.6 GB (FP16 ~15.2 GB). If still 15 GB, save_compressed=True failed. config.json must contain quantization_config with quant_method: "compressed-tensors", num_bits: 4, group_size: 128.

Quick load test with vllm.LLM; check quantization attribute shows expected method; generate sample outputs.

5.5 Server Launch Parameters

vllm serve ./out/Qwen2.5-7B-Instruct-AWQ-W4A16-G128 \
  --served-model-name qwen2.5-7b-awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --disable-log-requests

Each parameter explained: max-model-len sets KV cache budget per request (set to biz P99 + 20%); gpu-memory-utilization 0.90 leaves headroom for host sharing; max-num-seqs not larger is better — too high crowds KV space causing prefill queueing; max-num-batched-tokens affects chunked prefill granularity; enable-prefix-caching essential for multi-turn/fixed system prompt; disable-log-requests avoids disk pressure. No --quantization needed for compressed-tensors — auto-detected.

5.6 Go-Live Verification

Check /health, /v1/models, functional chat completion, Prometheus metrics ( vllm:kv_cache_usage_perc, num_requests_running, num_requests_waiting). Startup log must show model weights take 4.31 GiB, maximum concurrency for 8,192 tokens per request: 15.07x. If max concurrency < target concurrency, memory insufficient — reducing max-model-len or adding GPUs needed; quantization won't fix KV cache bottleneck.

Part 6: Practice 2 — GPTQ Quantization (GPTQModel Path)

6.1 Why Still Use GPTQ?

Legacy checkpoints on HuggingFace are GPTQ format; migration cost high.

ExLlamaV2 ecosystem (native GPTQ) faster there, though niche in 2026.

GPTQModel exposes more knobs ( damp, mse, per-layer config, act_group_aware) for models where AWQ struggles.

6.2 Installation & Adaptation

Separate venv, install ninja for JIT, gptqmodel, optionally gptqmodel[gguf]. Note class rename: AutoGPTQForCausalLM → GPTQModel; config class moved to gptqmodel.GPTQConfig.

6.3 Quantization Script with Checkpoint Resume

70B models take 3-6 hours; GPTQModel 1.12+ supports resume via offload_to_disk=True (Llama/Qwen3 MoE only). Uses allenai/c4 calibration. Key config: bits=4, group_size=128, damp=0.01, desc_act=False for Marlin, format=MARLIN for vLLM deployment.

6.4 Tuning Heuristics

desc_act

: vLLM+Marlin → False; ExLlamaV2/GPTQModel inference → True. group_size: 128 default; -1 (per-row) max compression but worst precision; 32/64 improves precision ~3-6% size increase. damp: default 0.01; increase to 0.05-0.1 if Cholesky fails/nan; decrease to 0.005 only if stable and precision critical. mse: grid search for optimal scale; 2-5× slower, gains 0.1-0.3 pp task metric — only if stuck at threshold.

Real lesson: 13B code model GPTQ ( desc_act=True, group_size=128, damp=0.01) dropped HumanEval 4.2 pts; adjusting to damp=0.05, mse=2.4, group_size=64 recovered to 1.8 pts but size 7.6→8.4 GB, time 40min→3hr. Switching to AWQ gave 0.9 pt drop in 18 min — algorithm fit matters more than tuning.

6.5 vLLM Loading GPTQ

Checkpoint has quant_method: "gptq"; auto-detected. Verify kernel: log should show Using MarlinLinearKernel. Explicit --quantization gptq_marlin forces kernel but may error if alias changed.

Part 7: Practice 3 — GGUF Generation & Usage

7.1 When to Use GGUF

Any of: (1) no NVIDIA GPU (CPU, Apple Silicon, ARM, iGPU); (2) single-node low concurrency (QPS single-digit); (3) need extreme portability (single file, no Python/CUDA). Not for GPU cluster high-throughput — vLLM GGUF plugin experimental, dequantizes to 16-bit, no Marlin/CUTLASS, lacks PagedAttention, some features incompatible.

7.2 Generate GGUF with llama.cpp

git clone https://github.com/ggml-org/llama.cpp
python convert_hf_to_gguf.py /models/Qwen2.5-7B-Instruct --outfile qwen2.5-7b-f16.gguf --outtype f16
cmake -B build -DGGML_CUDA=OFF && cmake --build build --config Release -j$(nproc)
./build/bin/llama-quantize qwen2.5-7b-f16.gguf qwen2.5-7b-Q4_K_M.gguf Q4_K_M 8

Don't skip FP16 intermediate — direct safetensors→Q4_K_M can cause precision issues. Output shows model size 15231 MB → quant size 4463 MB (~3.4× compression).

7.3 Export GGUF via GPTQModel

If GPTQModel env exists:

model = GPTQModel.load(..., GGUFConfig(bits=4, format="q_k_m")); model.quantize(calibration=None, backend=BACKEND.GGUF_TORCH); model.save(".gguf")

. Verify supported formats per version.

7.4 vLLM Load GGUF (Verification Only)

pip install vllm-gguf-plugin

;

vllm serve unsloth/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --tokenizer Qwen/Qwen2.5-7B-Instruct

. Must specify --tokenizer separately — GGUF tokenizer metadata conversion unreliable. Add --hf-config-path if config generation fails.

7.5 Native llama.cpp Deployment (GGUF's Proper Place)

cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=ON
./build/bin/llama-server --model qwen2.5-7b-Q4_K_M.gguf --alias qwen2.5-7b-gguf --ctx-size 8192 --n-predict -1 --threads $(nproc) --host 0.0.0.0 --port 8080 --metrics

Key: --threads = physical cores (hyperthreading low benefit); --metrics exposes Prometheus. Performance expectation: 7B Q4_K_M on 16-core x86 ~40-80 tok/s prompt, 8-15 tok/s generation — two orders magnitude below GPU, hardware-limited. For GPU offload: compile with -DGGML_CUDA=ON, use --n-gpu-layers 99.

Part 8: Performance Comparison — How to Get Credible Numbers

8.1 Fairness Principles

Control: same GPU, driver, CUDA; same base model & tokenizer; same vLLM version & params; same dataset (random or real samples); fixed input/output lengths, concurrency sweep; ≥3 runs per config, take median; 20-request warmup each run. Never load multiple quantized models in same process — allocator/CUDA graph state contaminates results.

8.2 Scripted Benchmark Matrix

Bash script loops over models, concurrency levels; each model gets fresh vLLM process; extracts metrics via embedded Python to handle JSON field name changes across vLLM versions.

8.3 Real Benchmark Results (7B on A100 80GB, input 512/out 256, random)

Format                Weight    Conc 1 tok/s  Conc 32 tok/s  Conc 32 TTFT P99  Conc 32 TPOT mean
BF16 (baseline)       15.2 GiB  58            1150           1180 ms           29 ms
AWQ W4A16 g128        4.4 GiB   96            2240           980 ms            14 ms
GPTQ 4bit + Marlin    4.5 GiB   92            2100           1010 ms           15 ms
GGUF Q4_K_M (vLLM)    4.5 GiB   41            860            1980 ms           38 ms
GGUF Q4_K_M (llama.cpp GPU) 4.5 GiB 55         640            1620 ms           34 ms

Interpretation: low concurrency gains from memory bandwidth reduction (1.6× not 3.5× due to KV cache, activation, kernel overhead). High concurrency gains larger because memory savings allow more concurrent requests. GGUF in vLLM suffers dequant path — quantization saves memory not compute. llama.cpp GPU better at concurrency 1 but falls further behind at high concurrency due to no PagedAttention. Numbers not portable across hardware — must run script on own GPUs.

8.4 Rigorous Accuracy Comparison

Rules: same eval script, prompts, seed; perplexity only auxiliary — task metrics primary; must test hard cases (long context >4K, exact numeric/format output, rare tokens, multi-step reasoning); report deltas vs baseline, not absolute scores. Provide compare_eval.py to diff lm_eval outputs.

Part 9: Selection Decision — Four-Step Framework

9.1 Step 1: Identify Bottleneck — Weight vs KV Cache

Calculate:

Total = Weight + KV cache + Activation peak + Framework overhead

. Example: Qwen2.5-7B (28 layers, 4 KV heads, head_dim 128), BF16 KV, 8192 ctx, 32 concurrency → KV cache ≈ 14 GiB. Weight BF16 ≈ 14.2 GiB. Total ~28 GiB + overhead → tight on A100 40G. Quantizing weight to 4-bit saves 10 GiB; reducing max_model_len to 4096 saves 7 GiB; FP8 KV cache saves 7 GiB. Priority: quantize KV first (least quality impact), then weight, lastly reduce context — but teams often do opposite.

9.2 Step 2: Confirm Hardware Kernel Support

Check compute capability: torch.cuda.get_device_capability(0). Marlin needs SM≥8.0 (Turing+); AWQ unsupported on Volta; INT8 W8A8 needs SM≥8.0; FP8 W8A8 needs SM≥8.9 (Ada/Hopper) — A100 (SM 8.0) cannot run FP8 ; GGUF runs everywhere.

9.3 Step 3: Choose Format by Scenario

GPU cluster, Ada/Hopper (SM≥8.9): high precision + 8-bit OK → FP8 W8A8; memory tight → AWQ W4A16 + Marlin
GPU cluster, Ampere (A100): memory OK → INT8 W8A8 or AWQ W4A16; memory tight → AWQ W4A16 + Marlin (preferred) or GPTQ+Marlin
GPU cluster, Turing (T4): AWQ or GPTQ (no FP8, Marlin OK)
Non-GPU: Apple Silicon/ARM/x86 CPU → GGUF (llama.cpp/Ollama); single consumer GPU → GGUF+llama.cpp or AWQ+vLLM
NPU (Ascend etc): vendor toolchain (ATC), GPTQ/AWQ not compatible

9.4 Step 4: Accuracy Remediation Order (Low to High Cost)

Reduce group_size (128→64/32): +3-6% size, +0.3-1 pt precision, 20 min requant.

Change calibration data to production distribution, increase samples 512→1024.

Switch algorithm (GPTQ↔AWQ).

Raise bit-width (W4→W8 INT8): 2× size, only 0.2-0.5 pt drop — often best ROI if memory allows.

Mixed precision: sensitive layers (MLP down_proj, attention o_proj) at 8-bit, rest 4-bit via config_groups.

Give up quantization: use larger GPU or smaller base model (distilled 7B often beats 72B 4-bit).

9.5 Mixed Precision Config Example

Use GPTQModifier with config_groups targeting re:.*down_proj and re:.*o_proj at 8-bit channel-wise, others 4-bit group-wise. Verify module names per model; memory savings diminish as 8-bit proportion grows.

Part 10: Performance Tuning — Squeezing More After Quantization

10.1 Kernel Selection: --linear-backend

vLLM 0.33 supports explicit backend: --linear-backend cutlass or per-quant scheme via

--kernel-config '{"linear_backend_per_quant":{"nvfp4_w4a16":"humming"}}'

. Needed when auto-pick chooses slow kernel, mixed precision requires specific backend, or debugging why slower than expected. Confirm via debug log: Using CutlassW4A8LinearKernel / Using MarlinLinearKernel.

10.2 Marlin Activation Precision Constraint (Breaking Change in 0.33)

Marlin now fixed to 16-bit activations; VLLM_MARLIN_INPUT_DTYPE removed (setting it errors). For W4A8 need humming quantization with VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"int8"}' or float8e4m3. Many 2025 guides still suggest old env var — causes startup failure.

10.3 KV Cache Quantization: Often Better ROI Than Weight Quantization

KV cache often exceeds weight memory in long-context. Quantizing KV (FP8) usually hurts quality less than weight quantization because errors don't accumulate across layers. Enable: --kv-cache-dtype fp8 (needs Ada/Hopper). Verify: startup log KV cache size should double (FP8 1 byte vs BF16 2 bytes). Risks: dynamic scale may degrade at context end; some attention backends silently fall back to 16-bit — check vllm:cache_config_info metric.

10.4 Scheduling Parameters Interaction with Quantization

--max-num-batched-tokens

can increase (quantization frees memory, faster weight load allows larger prefill batches).

Don't blindly raise --max-num-seqs — it pressures KV cache; quantization saves weight memory, not KV. Prefer using savings for larger max-model-len or concurrency headroom. --gpu-memory-utilization should be lowered post-quantization (e.g., 0.85-0.88) because activation peak becomes larger fraction of total.

10.5 Real Tuning Log (7B AWQ, A100 80G, concurrency 32)

Round  Change                              Output tok/s  TTFT P99  Observation
1      Default                             1650        1620 ms   Baseline
2      max-num-batched-tokens 8192→16384   1980        1180 ms   Larger prefill batches, shorter queue
3      enable-prefix-caching               2240        980 ms    System prompt hit rate 62%
4      max-model-len 32768→8192            2350        860 ms    Less KV reserved, more concurrency headroom
5      gpu-memory-utilization 0.95→0.88    2350        860 ms    Throughput same, OOMs disappeared
6      max-num-seqs 64→128                 2180        1420 ms   WORSE — KV saturation caused queueing
7      Revert to 64                        2350        860 ms    Confirmed 64 is knee point

Knee detection via PromQL: kv_cache_usage_perc > 0.95 + queue_time P95 > 1s + prefill_time P95 < 0.5s → KV shortage, not compute.

Part 11: Monitoring & Logging — What to Watch for Quantized Models

11.1 Core vLLM Metrics

Service-level: num_requests_running, num_requests_waiting, kv_cache_usage_perc (>0.95 for 5min → scale), prefix_cache_hits/queries, prompt_tokens_total / generation_tokens_total, request_success_total by finish_reason. Request-level (SLO): time_to_first_token_seconds (TTFT P95<1-2s), inter_token_latency_seconds (ITL P95<100ms), request_time_per_output_token_seconds (TPOT P95<60ms), e2e_request_latency_seconds, request_queue_time_seconds (P95<200ms), request_prefill_time_seconds, request_decode_time_seconds. Metric names change across versions — re-scrape /metrics after upgrade.

11.2 Alert Rules (Prometheus)

Alerts for KV cache near full, queue backlog, TTFT P99 high, TPOT P95 high, prefix cache hit rate low, generation truncated high. Thresholds must be calibrated to business baseline.

11.3 GPU-Side Monitoring (DCGM/nvidia_gpu_exporter)

Critical: DCGM_FI_PROF_PIPE_TENSOR_ACTIVE < 0.1 indicates quantization kernel not used (running dequant+FP16 GEMM slow path) — most common cause of "quantized but not faster". Also watch ECC errors, memory utilization, clock throttling, power violations.

Part 12: Common Failure Troubleshooting Paths

12.1 Load Failure: Unknown Quantization Method

Cause: vLLM too old for compressed-tensors or checkpoint quant_method unrecognized. Fix: upgrade vLLM, regenerate checkpoint, or avoid community forks.

12.2 Quantized But Memory Not Reduced

Check: file size (should be ~1/3.5), config.json has quantization config, safetensors dtype shows I32 / U8 with last dim ~1/8. Root causes: missing save_compressed=True, wrong load path, silent fallback (grep logs for fallback / not supported).

12.3 Throughput Drops After Quantization

Almost always kernel path issue. Debug: VLLM_LOGGING_LEVEL=DEBUG grep for kernel names; check Tensor Core activity via nvidia-smi dmon or DCGM; verify hardware compatibility (SM version). Common fixes: re-export as MARLIN format, explicit --quantization awq_marlin, ensure bits=4 and group_size in {32,64,128,256,-1}.

12.4 Output Quality Anomalies (Repetition, Garbage, Truncation)

Rule out chat template: run same checkpoint via transformers directly — if OK, vLLM template/tokenizer config issue.

Check eos_token_id / pad_token_id in generation_config.json and config.json — null or mismatch causes non-stop generation.

Suspect quantization damage: compare quantized vs FP16 outputs on same prompts; semantic collapse indicates wrong calibration data, missing ignore layers (embed_tokens), or nan during quantization.

12.5 OOM: Not All OOMs Are Weight Size

Distinguish CUDA OOM (vLLM error) vs System OOM (SIGKILL in dmesg). CUDA OOM sources: (1) weight+KV exceed gpu-memory-utilization → reduce max-model-len / max-num-seqs; (2) activation peak from large max-num-batched-tokens; (3) CUDA graph capture overhead → limit --cuda-graph-sizes or --enforce-eager. System OOM causes: quantization-time CPU memory peak (use offload_to_disk=True), safetensors double-copy peak, host sharing — container limit should be model need +30% buffer with host headroom.

12.6 Quantization Process Failures

CUDA OOM — Cause: Calibration seq too long/batch too large. Fix: Lower max_seq_length to 1024; set sequential_targets to block class name.

Cholesky decomposition failed — Cause: Hessian ill-conditioned. Fix: GPTQ: increase damp to 0.05-0.1.

nan in weights — Cause: Numerical instability. Fix: Increase damp; try sym=False; reduce group_size.

KeyError module name — Cause: Model structure vs recipe targets mismatch. Fix: Print actual module names, fix regex.

AssertionError dataset too small — Cause: Calibration samples < num_calibration_samples. Fix: Add samples or lower parameter.

General: run minimal config first ( num_calibration_samples=8, max_seq_length=512, single block) to validate pipeline before full run.

Part 13: Risk Assessment & Rollback Plan

13.1 Risk Register

Risks: accuracy shortfall (medium prob, high impact) → pre-deploy eval + canary + keep FP16 replica; specific input regression (high prob, high impact) → custom hard-case set + human spot-check; kernel fallback on some GPUs (medium) → pre-deploy stress test on same GPU type + log kernel confirmation; memory planning deviation (medium prob, high impact) → use measured logs not theory, leave buffer in limits; checkpoint corruption (low prob, high impact) → post-build load test + output diff + versioned storage; dependency drift (high prob, medium impact) → pin versions, CI smoke test for quantized models; calibration-distribution mismatch (medium-high) → use production samples, compare multiple calibration sets; rollback missing original model (low prob, high impact) → retain FP16 weights + full config + version record.

13.2 Rollback Preparation

Keep three artifacts: FP16 base, current quantized, previous quantized/alternative.

Freeze launch params in env file ( /etc/vllm/qwen7b.env) for one-command switch.

Rollback script: backup current env, swap MODEL_PATH to FP16, simultaneously reduce MAX_MODEL_LEN and MAX_NUM_SEQS (FP16 weight 3.5× larger — same params cause OOM), restart service, wait for health, verify with test prompt, keep backup for re-revert.

13.3 Canary Strategies

Preferred: shadow traffic — production requests → quantized replica (served), mirrored to FP16 replica (logged only). Compare outputs via semantic equivalence (strong model judge or business rule checks), not string similarity. Alternative: route low-risk tasks first (title candidates, tags) → structured extraction (cautious) → user-facing replies (last).

Part 14: Production Considerations

14.1 Version Pinning

Freeze pip freeze > requirements-*.lock for both quant and inference envs. Critical pins: vllm==0.33.0, torch==2.14.0, transformers, compressed-tensors (dictates checkpoint parsing), llmcompressor==0.15.0, gptqmodel. CI must include quantized model smoke test: load + 3 fixed prompts checking length, format, key tokens — not exact string match.

14.2 Storage & Distribution

Versioned registry: /models/registry/ModelName/scheme-date/ with quant_meta.json (crucial for later audits). Distribution: small scale → NFS/object store + local cache; large scale → bake into container image (5-30 GB, watch registry layer limits) or S3 via --model-loader-extra-config. Cold-start image pull often slower than model load.

14.3 GPU Memory & Multi-Replica Planning

Single GPU multi-process (split memory):

CUDA_VISIBLE_DEVICES=0 vllm serve ... --gpu-memory-utilization 0.45

— but processes contend for compute/bandwidth, 1+1<2; only for small models (<3B) or very low concurrency. MIG (A100/H100): hardware isolation, but smaller memory may not fit model, some kernels unsupported. Decision: if single replica maximum concurrency meets SLA and GPU utilization <30%, consider multi-replica; if >70%, add GPUs/nodes instead.

14.4 Re-Quantization on Base Model Upgrade

Cannot reuse quantization when base model changes (e.g., Qwen2.5→Qwen3): module names may change (new attention, MoE), tokenizer changes calibration token distribution, ignore list needs review, kernel support must be re-verified (head_dim, GQA alignment). Pre-quantization check script prints config and leaf module names; watch hidden_size % 128 — non-zero may force slower kernel or fail.

14.5 Cost Accounting

Monthly cost = GPU hourly rate × 720. Quantization effects: weight memory ÷3.5 → larger models or bigger KV cache; throughput ×1.5-2 → fewer replicas for same QPS. Example: 7B FP16 on 8×A100 @¥20k/card = ¥160k/mo. AWQ 1.9× throughput → 5 replicas = ¥100k/mo, saving ¥60k/mo. Quantization investment ~3 engineer-days + 20 GPU-hours → payback <1 week. But if QPS low (GPU utilization 5%), savings negligible — quantization value shifts to "run larger model" not "save money".

Part 15: Summary

Concepts: GPTQ/AWQ are algorithms, GGUF is a format. Compare on (algorithm, bit-width, group-size) tuple. GGUF Q4_K_M ≈4.8 bits, larger and often higher precision than AWQ 4-bit — "GGUF lower precision" is wrong comparison.

Selection: First calculate memory breakdown (weight vs KV), then check hardware kernel support, then choose format by deployment shape. Wrong order leads to "quantize harder" fallacy — A100 can't do FP8, Ascend can't run AWQ; hard constraints, not tunable.

Validation: No baseline = no science. Must measure FP16 memory/throughput/quality first. Quality: task metrics primary, perplexity secondary; custom hard-case set covering long-context, numeric/structured output, rare tokens, multi-step reasoning.

Performance: Quantization saves bandwidth, not compute. Low concurrency bandwidth-bound → clear gains; high concurrency prefill compute-bound → limited gains. GGUF in vLLM only saves memory (dequant path) — "quantized but not faster" → check kernel first.

Parameters: gpu-memory-utilization lower after quantization; max-num-seqs has knee point (beyond it TTFT worsens); max-num-batched-tokens can increase; KV cache quantization often better ROI than weight quantization.

Operations: Pin versions, keep FP16 base, write rollback script, CI smoke test. Rollback must adjust context/concurrency params simultaneously or rollback itself OOMs.

Boundaries: Accuracy remediation order: group_size → calibration data → switch algorithm → raise bit-width → mixed precision → smaller model/add GPU. "Distilled small model in FP16" often beats "large model at 4-bit" — engineering pragmatism over perceived sophistication.

Final truth: Quantization complexity lies not in quantization itself, but in validation and operations. Running oneshot takes ten minutes; proving quality, performance, stability, and rollback readiness is the real work.