LLM Inference Optimization: Metrics, Model & Service Techniques

1. Inference Performance Metrics

Performance metrics guide optimization direction from two perspectives:

User perspective : Latency — how fast the system responds.

Developer perspective : Throughput and Utilization — directly tied to Cost .

1.1 Latency Metrics

Latency determines interactive experience.

Time to First Token (TTFT) : Time from request to first output character. Key point : Depends on Prefill compute speed. Critical for search/chat apps because low TTFT makes users feel the system is "thinking" and responding quickly.

Time per Output Token (TPOT) : Average time per token after the first. Key point : Corresponds to Decode speed. TPOT defines the "streaming feel"; high TPOT causes perceived stutter and disjointed reading.

Total Latency = TTFT + (TPOT × Output Token Count)

1.2 Throughput Metrics

Throughput measures load capacity and processing efficiency.

Throughput (TPS) : Total tokens processed per second. Key point : Higher throughput means same hardware serves more concurrent users, lowering per-inference cost .

Goodput : Throughput under SLA/SLO constraints. Definition : Maximum tokens/sec while meeting latency targets (e.g., TTFT < 200ms, TPOT < 50ms). Significance : Pure high throughput with poor latency is useless; Goodput reflects usable capacity.

Note : Must distinguish Prefill throughput (input processing rate) and Decode throughput (generation rate) because their compute characteristics differ vastly.

1.3 Utilization Metrics

Utilization shows hardware "busyness" and reveals performance headroom.

Model FLOP/s Utilization (MFU) : Actual FLOPs vs. hardware peak. Current state : Inference MFU is typically far lower than training due to memory bandwidth bottlenecks.

Model Bandwidth Utilization (MBU) : Used memory bandwidth vs. peak bandwidth. Significance : LLM inference (especially Decode) is usually Memory-bound , making MBU a more critical efficiency indicator.

Summary : High utilization is a means; the end goal is best experience (low latency) at lowest cost (high throughput) .

2. Model-Level Optimizations

These modify model structure or computation to reduce compute and memory footprint.

2.1 Model Compression

Quantization : Reduce weight/activation precision from FP16/BF16 to INT8, INT4, or FP4. Principle : Algorithms like GPTQ, AWQ preserve model quality despite precision loss. Benefit : Halves or further reduces memory usage; leverages integer compute units for speed — currently the highest ROI optimization.

Distillation : Train a smaller model to mimic a large pre-trained model. Benefit : Smaller models naturally run faster, suitable for edge/low-compute deployment.

Pruning : Remove near-zero-impact parameters (zero-out or delete). Sparsification : Pruned models become sparse, requiring specialized sparse kernels to realize acceleration.

2.2 Autoregressive Decoding Optimizations

LLM's token-by-token generation causes massive memory read/write overhead. These techniques break the serial bottleneck:

Speculative Decoding : Mechanism : A tiny Draft Model quickly generates a candidate sequence (e.g., 5 tokens), then the large model verifies all candidates in parallel. Advantage : If draft guesses correctly, the large model skips serial steps; parallel verification is much faster than serial generation.

Prompt Lookup Decoding : Mechanism : Similar to RAG but at decode time. If current context appeared in the input prompt, copy the subsequent segment as candidates instead of model "thinking". Scenario : Excels in code completion, document summarization where input-output overlap is high.

Parallel Decoding : Example : Medusa architecture adds multiple prediction heads, letting the model predict token N+1, N+2 while predicting token N.

2.3 Attention Mechanism Optimizations

Transformer attention is the compute core and long-context bottleneck.

KV Cache : Principle : Keys and Values for past tokens are fixed during decoding; cache them to avoid recomputing history each step. Cost : KV Cache consumes large GPU memory, growing linearly (or super-linearly) with sequence length — the main limiter of Context Window.

Optimizing Attention Compute :

FlashAttention : IO-aware algorithm that drastically reduces HBM (High Bandwidth Memory) accesses, significantly speeding up long-context computation.

Structural Optimizations : GQA (Grouped Query Attention) and MLA (Multi-head Latent Attention) reduce Key/Value count (i.e., KV Cache size) while preserving performance.

KV Cache Management :

PagedAttention (vLLM) : Borrows OS virtual memory/paging ideas, splitting KV Cache into non-contiguous blocks. Solves memory fragmentation, greatly improves memory utilization, enabling larger batch sizes.

3. Inference Service Optimizations

These optimize request scheduling, resource allocation, and system architecture without changing the model.

3.1 Batching

GPU parallelism requires "batching" requests.

Static Batching : Wait for N requests (or timeout) then process together. Drawback : "Bucket effect" — batch time equals slowest request; short requests wait idle, wasting compute and increasing latency.

Continuous Batching (In-flight Batching) : Standard in modern engines (vLLM, TGI, TRT-LLM). Mechanism : Schedules at iteration level , not request level. When a request finishes (hits EOS), it's immediately removed and a new request inserted into the current batch. Advantage : Keeps GPU fully utilized, eliminates idle wait, dramatically boosts throughput.

3.2 Prefill/Decode Separation (Splitwise Architecture)

Background :

Prefill is Compute-bound — needs massive compute to process prompt quickly.

Decode is Memory-bandwidth-bound — low compute utilization, bottlenecked on moving KV Cache.

Optimization : Split stages onto separate GPU instances.

Prefill instances : Focus on fast input processing, compute KV Cache, transfer to Decode instances.

Decode instances : Receive KV Cache, perform streaming generation.

Benefit : Eliminates resource contention (e.g., memory bandwidth) between stages, significantly lowers TTFT and raises overall throughput.

3.3 Prompt Caching

Principle : Many requests share common prefixes (e.g., long document in QA, Agent system prompts).

Mechanism : Engine uses Radix Tree or similar to detect shared prefixes and reuse previously computed KV Cache.

Benefit :

Repeated system prompts or multi-turn history: Prefill time approaches zero.

RAG and long-document analysis: Drastically reduces first-token latency and compute cost.

3.4 Parallelism Strategies

When a single GPU can't hold the model or meet performance, multi-GPU coordination is needed.

Tensor Parallelism (TP) : Split : Matrix multiplications "horizontally" across GPUs. Trait : Frequent communication; typically used within a single node (NVLink), effectively reduces per-request latency .

Pipeline Parallelism (PP) : Split : Model layers "vertically" across devices. Trait : Suits cross-node deployment; mainly boosts throughput , limited impact on single-request latency.

Data Parallelism / Replica Parallelism : Method : Launch multiple full model replicas. Trait : Simplest horizontal scaling, purely increases concurrent capacity (QPS).

4. Summary

Inference optimization is a systems engineering effort:

Pick the right metrics : Clarify whether targeting ultra-low latency or high-throughput low-cost.

Combine software and hardware : Model-level quantization/structural optimizations must pair with service-level continuous batching, KV Cache management to maximize effect.

Adapt to scenario : Long-context focuses on KV Cache and Attention; high-concurrency focuses on Batching and scheduling strategies.