Essential Concepts and Terminology for Deploying Large Language Models Locally

1. Weight Precision

Model weights are stored in different numeric formats, each affecting size and compute cost.

FP32 – 32‑bit (8‑bit exponent, 23‑bit mantissa). Highest accuracy, largest memory footprint.

FP16 (F16) – 16‑bit (5‑bit exponent, 10‑bit mantissa). About half the size of FP32, common for inference; limited exponent range can cause overflow in some layers.

BF16 (bfloat16) – 16‑bit (8‑bit exponent, 7‑bit mantissa). Same dynamic range as FP32, slightly more rounding error than FP16.

FP8 (E4M3/E5M2) – 8‑bit (4‑ or 5‑bit exponent, 3‑ or 2‑bit mantissa). Each parameter occupies 1 byte, comparable to INT8; hardware support varies.

Rough size estimate: a 7 B model in FP16 occupies ~14 GB; FP8 reduces this roughly by half. These numbers exclude KV cache, activations, and runtime overhead.

1.2 Model Architecture Variants

Two families:

Dense models – no active‑expert marker; all parameters are used each forward pass.

Mixture‑of‑Experts (MoE) – marked with an A tag such as 14B‑A3B, indicating total parameters and active parameters per forward.

KV‑cache size depends heavily on the attention architecture. Using MHA formulas for GQA/MLA will over‑estimate memory.

2. Weight Quantization

Quantization compresses high‑precision weights to lower bit‑widths, reducing storage and memory.

2.1 Non‑K‑Series Quantization

Q8_0 – ~8‑bit, about 50 % of FP16 size; common in the llama.cpp ecosystem.

FP8 (W8A8) – 8‑bit weight and activation; ~50 % of FP16 size, requires hardware and engine support.

FP8 (W8A16) – 8‑bit weight, 16‑bit activation; similar size to W8A8 with better compatibility.

2.2 K‑Quant (llama.cpp) Series

Quantization schemes embed the bit‑width in the filename suffix ( _K) and optionally a quality hint ( _M for medium, _S for small):

Q6_K – mixed‑precision 6‑bit, ~42 % of FP16 size, ~5.5 GB for a 7 B model.

Q5_K_M – medium 5‑bit, ~37 % of FP16 size, ~4.8 GB.

Q5_K_S – small 5‑bit, ~35 % of FP16 size, ~4.6 GB; slightly lower quality than _M.

Q4_K_M – medium 4‑bit, ~32 % of FP16 size, ~4.1 GB; a common balanced choice.

Q4_K_S – small 4‑bit, ~29 % of FP16 size, ~3.8 GB; often used when VRAM is tight.

These figures are approximate; actual quality must be validated on the target task.

3. Packaging Formats

After quantization, weights are packaged for loading by inference engines. Common formats include:

GGUF – single‑file .gguf, used by llama.cpp and Ollama; supports CPU, CUDA, Metal, SYCL, etc.

GPTQ – directory of .safetensors shards; primarily CUDA‑based.

AWQ – also .safetensors shards; activation‑aware quantization, NVIDIA‑centric.

EXL2 – .safetensors shards; native to ExLlamaV2.

MLX – directory format for Apple Silicon.

NF4 (bitsandbytes) – directory or load‑time format; 4‑bit NormalFloat, often paired with QLoRA.

compressed‑tensors – generic directory for quantized or sparse tensors, used by vLLM and similar engines.

4. Inference Engines

Engines load weights, manage memory, and perform token decoding. Common options:

llama.cpp – wide hardware support, memory‑mapped loading, layer off‑loading.

Ollama – focuses on ease of model management, fewer low‑level controls.

ExLlamaV2 – CUDA‑only, optimized for single‑sequence latency, native EXL2 format.

vLLM – extreme concurrency via PagedAttention, supports many formats (FP16, BF16, FP8, AWQ, GPTQ, compressed‑tensors).

SGLang – similar to vLLM with high concurrency, uses prefix KV reuse (RadixAttention).

TGI – Hugging Face inference service, container‑friendly, supports CUDA/ROCm.

MLX‑LM – optimized for Apple Silicon, lower concurrency.

LMDeploy – CUDA‑based, includes TurboMind backend.

MLC‑LLM – cross‑platform (CUDA, Metal, Vulkan, WebGPU), low concurrency, custom quantizations.

4.2 Speculative Decoding

To reduce per‑token cost, engines may generate draft tokens with a lightweight path and verify them with the full model. Terminology varies by engine:

MTP (Multi‑Token Prediction) – model predicts multiple future tokens in one pass.

DFlash – lightweight draft structure that avoids a separate draft model.

DSpark – schedules draft generation, balancing cost, quality, and acceptance rate.

Effectiveness depends on the specific algorithm and implementation; always consult engine documentation and empirical results.

5. Runtime Memory (VRAM)

Beyond weights, memory is consumed by KV‑cache, activations, and driver/framework buffers.

5.1 KV Cache

During autoregressive generation each layer caches Key/Value pairs to avoid recomputation. Approximate size for a single sequence:

KV Cache ≈ 2 × L × KV_Heads × Head_Dim × T × B

where L = number of layers, T = context length, B = bytes per element (e.g., 2 for FP16). For MHA, KV_Heads × Head_Dim = Hidden_Size. GQA and MQA require the actual KV_Heads value; MLA uses a model‑specific compressed dimension.

5.2 Context Length

The maximum token window is limited by positional encodings and training length. Engines expose parameters such as num_ctx, -c, or max_model_len. Allocation strategies differ:

Static pre‑allocation (e.g., llama.cpp, Ollama) reserves the full upper bound regardless of actual usage.

Dynamic paging (e.g., vLLM, SGLang) allocates on demand up to a safety margin.

5.3 Long‑Context Deployment Tips

Estimate KV memory for the target length; it can approach or exceed weight memory.

KV quantization (8‑bit or 4‑bit) can save VRAM but must be validated for quality.

Prefer native context lengths; extrapolation may increase window size but not guarantee quality.

Match the engine’s allocation policy to avoid over‑reservation.

6. Sampling Strategies

Sampling determines how the next token is chosen from the model’s probability distribution. Parameters are lightweight but affect determinism and diversity:

Temperature – divides logits by T before softmax; higher T yields flatter, more random distributions; T →0 approximates greedy selection.

Top‑p (nucleus) – keeps the smallest set of tokens whose cumulative probability exceeds p (commonly 0.9) and renormalizes.

Top‑k – retains the top k tokens by probability before sampling; often applied before Top‑p.

Repeat Penalty – reduces the probability of tokens that have already appeared; exact semantics differ per engine (e.g., repeat_penalty vs frequency_penalty).

Min‑p – discards tokens whose probability is below min_p × max_prob; useful for eliminating very low‑probability tails.

7. Functional Extensions

Typical add‑ons beyond the core language model:

Embedding models – map text to vectors for retrieval; examples: BGE, Nomic, Jina. Consume separate VRAM and can be quantized independently.

LoRA adapters – low‑rank fine‑tuning weights that modify model behavior; must be loaded on top of the base model.

RAG pipelines – retrieve → augment context → generate; implementations: AnythingLLM, Dify, LangChain.

Vector databases – high‑dimensional nearest‑neighbor search; e.g., Chroma, Qdrant, Milvus. Vector dimension must match the embedding model.

MCP (Model‑Control‑Protocol) – defines how a model talks to external tools or data sources; typically requires a separate server process.

Function calling – structured JSON‑like output that triggers external functions; distinct from guided decoding but can be combined.

8. Deployment Decision Workflow

Practical engineering sequence:

Define hardware and scenario : total VRAM, CPU off‑loading, concurrency needs, availability of FP8 paths.

Select an engine based on the primary goal:

High‑concurrency API – start with vLLM or SGLang .

Low‑latency single‑GPU – ExLlamaV2 (EXL2).

Resource‑constrained / Apple Silicon – llama.cpp (GGUF) or MLX‑LM .

Enterprise container deployment – TGI or vLLM .

Back‑track quantization and format : choose the highest‑quality quantization the engine supports; if VRAM is insufficient, consider FP8, then AWQ/GPTQ, then more aggressive GGUF/EXL2 paths.

Set context length and KV precision : match the task’s required window; apply KV quantization (8‑bit/4‑bit) only after validation.

Configure sampling : low temperature for deterministic tasks, moderate temperature with Top‑p (≈0.9) for creative generation; be mindful of Top‑k + Top‑p interactions and engine‑specific repeat penalty semantics.

Account for extensions : add embedding, LoRA, RAG, etc., to the VRAM budget; if over budget, further quantize or off‑load these components.

Iterate and validate : measure actual VRAM usage using the formula

Total VRAM ≈ Weights + KV_Cache + Activations + CUDA_Overhead + Framework_Buffer

and adjust quantization, context, or engine choice until the model runs reliably.

When out‑of‑memory (OOM) occurs, revisit the hierarchy: lower weight quantization, shrink context, enable expert off‑loading, or select a more memory‑efficient engine.