How to Choose Between INT8, FP8, and INT4 Quantization for Large Models

Overview

Deploying a large model with different numeric precisions—INT8, FP8, or INT4—can lead to vastly different memory usage, throughput, latency, and answer quality. A common operational mistake is to decide solely based on model file size or the claim that "4‑bit saves memory" without considering weight precision, activation precision, KV‑Cache format, compute kernels, and hardware support. This can result in a model that loads but runs slower, exhausts KV‑Cache, or degrades capabilities such as code generation, mathematics, tool calling, and long‑context reasoning.

Why Quantization Choice Is a Full Deployment Decision

The choice of quantization is not only a model‑level issue; it involves the entire deployment stack:

Model architecture determines weight size and KV‑Cache dimensions.

Hardware determines which low‑precision operators actually accelerate inference.

Inference framework determines whether the quantization format is supported.

Business traffic determines whether first‑token latency or batch throughput is more important.

Quality requirements determine how much loss can be tolerated.

This article targets NVIDIA GPU environments with common open‑source inference frameworks and emphasizes engineering differences, verification methods, validation scripts, gray‑release, and rollback procedures. Framework parameters change across versions; always verify the local version before running any command.

Key Placeholders

<model_dir>

: Local model directory or approved model identifier. <model_name>: Service name for the model. <GPU_id>: Target GPU index, e.g., 0 or 0,1. <API_path>: OpenAI‑compatible endpoint root (without trailing slash). <API_key>: Access token (never store in repo or shell history). <prompt_file>: De‑identified local JSONL test set. <output_dir>: Directory for test results with sufficient capacity and access control.

1. Clarify the Bit‑Width Terminology

Before any review, answer the following questions about the model:

What precision are the weights (e.g., W8, W4, FP8)?

What precision are the activations (e.g., A16, A8, FP8)?

Does matrix multiplication use native low‑precision Tensor Cores or does it de‑quantize to FP16/BF16 first?

What is the accumulation precision (most kernels do not use 4‑bit accumulation)?

What format is the KV‑Cache (FP16/BF16, FP8, or other)?

Is the quantization static (offline calibration) or dynamic (at load time)?

Which layers keep high precision (e.g., embedding, lm_head, normalization, or outlier channels)?

Only after these details are clear can memory and performance comparisons be meaningful.

1.1 INT8

INT8 stores values as 8‑bit integers, usually with a scale and sometimes a zero‑point. Common paths:

W8A16 : Weights 8‑bit, activations remain FP16/BF16. Saves weight memory; speed depends on kernel support.

W8A8 : Both weights and activations are 8‑bit; usually requires calibration or smoothing of outlier channels.

Partial‑linear‑layer quantization : Sensitive layers stay at higher precision to control quality loss.

INT8 is mature, works on hardware lacking native FP8 support, and is suitable when quality is critical but BF16 memory is insufficient.

1.2 FP8

FP8 is an 8‑bit floating‑point format. Two common encodings are E4M3 (more mantissa bits) and E5M2 (larger exponent range). The actual format used for weights, activations, and KV‑Cache depends on the model, hardware, and framework.

Advantages of FP8 on GPUs with native FP8 Tensor Cores:

Reduces weight/activation bandwidth while increasing compute throughput.

Handles large‑model outliers and dynamic range differently from INT8.

However, "GPU supports FP8" does not guarantee the model runs efficiently. Verify driver, CUDA, PyTorch, and framework versions, ensure the model architecture has FP8 kernels (especially for MoE, GQA, custom attention, or custom ops), and check for fallback operators that could degrade performance.

1.3 INT4

INT4 usually refers to 4‑bit weight quantization (e.g., W4A16 or W4A8). Weights are packed; during inference they are unpacked and either de‑quantized to FP16/BF16 or processed directly with specialized kernels.

Typical deployment formats include AWQ and GPTQ, but implementations differ in group size, symmetry, and version compatibility.

INT4 offers the greatest memory savings, making it attractive for memory‑constrained or high‑density single‑GPU deployments, but it brings higher quality risk:

More sensitive to quantization algorithm, calibration data, and group size.

Small batch sizes may see unpacking overhead outweigh bandwidth gains.

Some hardware lacks efficient W4 kernels, causing slower inference than INT8/FP8.

Mathematics, code, long‑context, rare languages, and structured output are more likely to expose quality loss.

Framework support for AWQ/GPTQ varies.

NF4 (bitsandbytes 4‑bit floating format) is often used for low‑memory training but is not equivalent to production INT4 inference formats.

2. Memory Breakdown Beyond Weights

Memory consumption consists of five parts, not just raw weight size:

Weight memory : Approximate lower bounds—BF16/BF16: 2 bytes per parameter; FP8/INT8: 1 byte; INT4: 0.5 byte. For a 27 B‑parameter model, the theoretical minima are 54 GB (BF16), 27 GB (FP8/INT8), and 13.5 GB (INT4). Actual nvidia‑smi usage is higher due to scale tensors, quantization metadata, unquantized layers, CUDA context, temporary workspace, and fragmentation.

KV‑Cache : For autoregressive generation, each layer stores Key and Value tensors. Approximate per‑token bytes ≈ 2 × layers × KV‑heads × head_dim × element_size. The cache scales linearly with batch size, sequence length, and number of concurrent requests. FP8 weight models may appear memory‑light for short requests but still OOM on long‑context high‑concurrency workloads because KV‑Cache remains in FP16/BF16 unless explicitly quantized.

Activations & Temporary Workspace : Prefill of long prompts, large batches, CUDA Graph, MoE routing, all‑reduce, and quantization kernels may allocate additional temporary memory. Some frameworks pre‑allocate a large KV‑Cache pool, so nvidia‑smi may show near‑full usage even without a leak.

Runtime & Communication Overhead : Tensor Parallel splits weights across GPUs, but each GPU still holds CUDA context, NCCL buffers, activations, and partial copies. Multi‑GPU total memory is not simply divided by the number of GPUs; inter‑GPU communication can become the new bottleneck after quantization speeds up compute.

Host Memory & Model Load Peak : Offline quantization may create high‑precision temporary tensors before discarding the original BF16 weights. Ensure sufficient RAM, shared memory, temporary directories, and disk space. A model that fits in 80 GB GPU memory does not guarantee it can be loaded with only 64 GB system RAM.

3. Engineering Comparison of the Three Schemes

Key dimensions (summarized from the original table):

Original weight bit‑width : INT8/FP8 = 8 bit, INT4 = 4 bit.

Typical paths : INT8 – W8A16 / W8A8; FP8 – W8A8 or FP8 checkpoint; INT4 – W4A16 / W4A8 (AWQ, GPTQ).

Memory savings : INT8 – moderate; FP8 – moderate; INT4 – maximum.

Quality risk : INT8 – low to moderate; FP8 – low to moderate; INT4 – moderate to high, model‑dependent.

Calibration dependence : INT8 (W8A8) often needs calibration; FP8 may be static or dynamic; INT4 (AWQ/GPTQ) usually requires calibration.

Hardware dependence : INT8 – mature kernels; FP8 – native FP8 hardware and framework support; INT4 – efficient W4 kernels and format support.

Latency (small batch) : INT8 – needs measurement; FP8 – advantage when supported; INT4 – may suffer unpack overhead.

Throughput (large batch) : INT8 – often better; FP8 – usually better when supported; INT4 – depends on kernel, memory bandwidth, and scheduler.

Format compatibility risk : INT8 – medium; FP8 – medium; INT4 – high.

Preferred scenarios : INT8 – robust compression, legacy paths; FP8 – high‑performance on modern data‑center GPUs; INT4 – strict memory constraints, single‑GPU high‑density deployments.

All quality claims must be validated against a BF16/FP16 baseline using the same data, prompts, sampling parameters, and hardware.

4. When INT8 Is Preferable

GPU/framework INT8 path is more mature than FP8.

Business requires high quality and wants to start with 8‑bit compression.

Existing, validated W8A8 calibration pipeline is available.

Model or specific operators lack FP8 support but have INT8 kernels.

Cross‑hardware or CPU deployment needs a unified quantization strategy.

5. When FP8 Is Preferable

Running on data‑center GPUs with native FP8 support (e.g., NVIDIA B200/B300).

Framework provides mature FP8 kernels for the model architecture.

Goal is to keep weight memory similar to INT8 while gaining higher throughput.

BF16 model runs but KV‑Cache dominates memory; FP8 can reduce KV‑Cache size if configured.

6. When INT4 Is Worth It

Weight memory (INT8/FP8) does not fit the target GPU.

Need to deploy a larger model on a single GPU to avoid cross‑GPU communication.

Business accepts systematic evaluation in exchange for higher deployment density.

Framework supports the specific AWQ/GPTQ checkpoint and quantization configuration.

Workloads are medium‑small batch and the kernel’s unpack overhead does not outweigh bandwidth gains.

7. Practical Verification Steps

7.1 Environment Baseline

Record GPU and software environment before any change:

nvidia-smi --query-gpu=index,name,uuid,driver_version,memory.total \
    --format=csv,noheader
nvidia-smi topo -m
python3 --version

Check installed Python packages:

python3 - <<PY
import json, platform, torch
print({
    "python": platform.python_version(),
    "torch": torch.__version__,
    "torch_cuda": torch.version.cuda,
    "cuda_available": torch.cuda.is_available(),
    "device_count": torch.cuda.device_count()
})
PY

Verify versions of transformers, accelerate, bitsandbytes, vllm, torchao and lock them in a virtual environment.

7.2 Model Format Inspection

Never trust the directory name. Inspect config.json for:

jq '{model_type, architectures, torch_dtype, quantization_config}' <model_dir>/config.json

List file sizes:

du -sh <model_dir>
find <model_dir> -maxdepth 1 -type f -printf '%f	%s bytes
' | sort

If the model uses safetensors, you can read only the header:

python3 - <<PY
from pathlib import Path
from safetensors import safe_open
model_dir = Path("<model_dir>")
for path in sorted(model_dir.glob("*.safetensors")):
    with safe_open(path, framework="pt", device="cpu") as handle:
        for key in handle.keys():
            tensor = handle.get_slice(key)
            print(key, tensor.get_dtype())
PY

Check quantization_config and loading logs to confirm the actual compute precision.

7.3 Validation Scripts

Example for loading INT8 with bitsandbytes:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_path = "<model_dir>"
quant_cfg = BitsAndBytesConfig(load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype=torch.float16,
    quantization_config=quant_cfg,
    trust_remote_code=False,
)
inputs = tokenizer("请简要说明 TCP 三次握手。", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

For FP8 with vllm (if the binary lists --quantization fp8 in --help), start the service:

CUDA_VISIBLE_DEVICES=<GPU_id> \
    vllm serve <model_dir> \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name <model_name> \
    --dtype auto \
    --quantization fp8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.85

For INT4 via AWQ:

CUDA_VISIBLE_DEVICES=<GPU_id> \
    vllm serve <model_dir> \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name <model_name> \
    --quantization awq \
    --dtype half \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.85

For INT4 via GPTQ, replace --quantization gptq.

7.4 Quality Evaluation

Use BF16/FP16 as the baseline. Run the same benchmark suite for each candidate under identical conditions (same model version, tokenizer, GPU, driver, framework version, tensor/pipeline parallelism, input token distribution, output limits, sampling parameters, warm‑up, and concurrency). Include the following sub‑benchmarks:

Mathematics reasoning (accuracy, stability, long‑chain errors).

Code generation (compilable rate, unit‑test pass rate, syntax correctness).

Multilingual capability (Chinese, English, domain‑specific language).

Tool calling (JSON schema compliance, parameter types, required fields).

Structured output (JSON, YAML, SQL adherence).

Long‑context retrieval (fact recall at different positions, cross‑paragraph reasoning, prompt injection boundaries).

Domain‑specific tasks (business terminology, retrieval‑augmented answers, safety refusals).

Generation stability (repetition, garbled text, premature EOS).

Report not only average scores but also per‑category differences, confidence intervals, and failure cases.

7.5 Performance Baseline

Fix all test conditions (model, tokenizer, GPU, power mode, driver, framework, parallelism, concurrency ladder, input token distribution, max output, prefill/decode settings, CUDA Graph, prefix cache). Record per‑request latency, TTFT, end‑to‑end latency, tokens‑per‑second, GPU utilization, power, temperature, and OOM events.

Example script that logs GPU metrics every second:

#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="<output_dir>"
GPU_QUERY_FILE="${OUTPUT_DIR}/gpu-metrics.csv"
mkdir -p "${OUTPUT_DIR}"
nvidia-smi \
  --query-gpu=timestamp,index,name,memory.used,memory.total,utilization.gpu,power.draw,temperature.gpu \
  --format=csv \
  --loop=1 \
  > "${GPU_QUERY_FILE}"

Example Python load‑test script (non‑streaming) that measures latency, success rate, and completion tokens. It reads a JSONL file of prompts, sends concurrent requests, and outputs a JSON summary with percentiles.

#!/usr/bin/env python3
import argparse, concurrent.futures, json, statistics, time, urllib.request, urllib.error

def percentile(values, fraction):
    if not values:
        return None
    ordered = sorted(values)
    idx = min(len(ordered)-1, int((len(ordered)-1)*fraction))
    return ordered[idx]

def request_once(api_url, api_key, model, prompt, timeout):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 256,
    }).encode("utf-8")
    req = urllib.request.Request(f"{api_url.rstrip('/')}/v1/chat/completions",
                                 data=body,
                                 headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
                                 method="POST")
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            payload = json.loads(resp.read())
            elapsed = time.perf_counter() - start
            usage = payload.get("usage", {})
            return {"ok": True, "latency": elapsed, "completion_tokens": int(usage.get("completion_tokens", 0))}
    except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as e:
        return {"ok": False, "latency": time.perf_counter() - start, "error": repr(e)}

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--api-url", required=True)
    parser.add_argument("--api-key", required=True)
    parser.add_argument("--model", required=True)
    parser.add_argument("--prompts", required=True)
    parser.add_argument("--concurrency", type=int, default=4)
    parser.add_argument("--timeout", type=float, default=120.0)
    args = parser.parse_args()
    prompts = []
    with open(args.prompts, "r", encoding="utf-8") as f:
        for i, line in enumerate(f, 1):
            if not line.strip():
                continue
            item = json.loads(line)
            if not isinstance(item.get("prompt"), str):
                raise ValueError(f"line {i}: missing string prompt")
            prompts.append(item["prompt"])
    if not prompts:
        raise ValueError("no prompts loaded")
    wall_start = time.perf_counter()
    with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as ex:
        futures = [ex.submit(request_once, args.api_url, args.api_key, args.model, p, args.timeout) for p in prompts]
        results = [f.result() for f in futures]
    wall_time = time.perf_counter() - wall_start
    succeeded = [r for r in results if r["ok"]]
    failed = [r for r in results if not r["ok"]]
    latencies = [r["latency"] for r in succeeded]
    completion_tokens = sum(r["completion_tokens"] for r in succeeded)
    summary = {
        "requests": len(results),
        "succeeded": len(succeeded),
        "failed": len(failed),
        "wall_time_seconds": wall_time,
        "requests_per_second": len(succeeded)/wall_time if wall_time else None,
        "completion_tokens": completion_tokens,
        "completion_tokens_per_second": completion_tokens/wall_time if wall_time else None,
        "latency_mean_seconds": statistics.mean(latencies) if latencies else None,
        "latency_p50_seconds": percentile(latencies, 0.50),
        "latency_p95_seconds": percentile(latencies, 0.95),
        "errors": [r.get("error") for r in failed[:10]],
    }
    print(json.dumps(summary, ensure_ascii=False, indent=2))

if __name__ == "__main__":
    main()

Run the script with appropriate arguments and record the wall‑clock throughput, latency percentiles, and error rates.

7.6 Stair‑case Concurrency Testing

Test concurrency levels 1, 2, 4, 8, 16, etc., until any of the following limits are reached:

P95/P99 latency exceeds SLO.

Error or timeout rate rises.

KV‑Cache shortage causes scheduling stalls.

GPU utilization saturates without further throughput gain.

Power, temperature, or host CPU/memory limits are hit.

Warm up before each step; exclude kernel compilation, CUDA Graph capture, and cold‑start from steady‑state measurements.

7.7 KV‑Cache Quantization

FP8 weight quantization does not imply FP8 KV‑Cache. If the framework supports --kv-cache-dtype fp8, test it separately:

vllm serve <model_dir> \
  --host 127.0.0.1 \
  --port 8000 \
  --served-model-name <model_name> \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85

Validate long‑context performance, fact recall, and attention stability; do not assume KV‑Cache follows weight precision.

8. Deployment Checklist (Blue‑Green Strategy)

Switching model services is a high‑risk change. Keep the existing BF16/FP8 instance (blue) while deploying the new quantized instance (green). Follow these steps:

Record model checksum, image digest, configuration, and dependency versions.

Verify GPU has enough free memory and power headroom.

Run de‑identified regression tests and stair‑case concurrency on the green instance.

Expose the green instance on an isolated port or node; do not receive production traffic yet.

Gradually shift traffic: 1 % internal, then 5 %, 10 %, 25 %, 50 %, 100 % while monitoring error rate, latency percentiles, TTFT, TPOT/ITL, KV‑Cache usage, GPU OOM, and process restarts.

Maintain the blue environment for a defined rollback window.

If any metric exceeds thresholds, stop the rollout, cut traffic back to blue, and investigate.

After successful rollout, archive logs, configurations, and model checksums for audit.

9. Common Pitfalls

Equating file size with GPU memory usage (ignores scale tensors, KV‑Cache, workspace).

Assuming FP8 weight quantization also quantizes KV‑Cache.

Testing only whether the model can answer a question (no quality data, no performance ladder).

Testing only concurrency 1 (does not reveal batch or KV‑Cache pressure).

Comparing under inconsistent conditions (different context length, batch size, sampling).

Trusting directory naming instead of inspecting quantization_config.

Ignoring kernel fallback (model may load but run high‑precision kernels).

Relying on overall average scores that hide domain‑specific regressions.

Overwriting the existing service without a fast rollback path.

Copy‑pasting parameters from another version after framework upgrades (quantization methods may change).

10. Final Acceptance Checklist

Before deployment

GPU, driver, CUDA, PyTorch, inference framework, and container digest recorded.

Model config, quantization config, checksum, and license verified.

Weight, KV‑Cache, workspace, and concurrency memory budgets calculated.

BF16/FP16 baseline quality and performance established.

Hardware supports the target quantization kernel (not just loadability).

Calibration data de‑identified and covers real traffic distribution.

INT8, FP8, and INT4 candidates completed stair‑case load tests under identical conditions.

Quality evaluation covers mathematics, code, tool calling, multilingual, and long‑context scenarios.

During rollout

New instance runs on an isolated port or node.

Health checks and fixed‑prompt regression run before any traffic.

Gradual gray release with continuous monitoring of error rate, TTFT, TPOT/ITL, throughput, KV‑Cache, and GPU metrics.

Blue environment remains healthy and can be switched back instantly.

After rollout

Results, configurations, model checksum, and test data archived.

Peak traffic meets SLOs without sustained OOM or restarts.

Business‑level quality checks (structured output error rate, etc.) are within targets.

Original model and rollback images retained for the agreed retention period.

Future framework or driver upgrades trigger a fresh compatibility and regression test cycle.