How GPUs Power LLM Inference: Compute, VRAM, Bandwidth & Frameworks Explained

When people first run large language models locally, they often wonder why discussions focus on GPUs rather than CPUs. Both can compute, but they excel at different tasks. CPUs have fewer, versatile cores suited for OS tasks and complex logic; GPUs have thousands of parallel units ideal for repeating the same operation — exactly what LLM inference demands: massive matrix multiplications and multiply-accumulate operations that are large, uniform, and parallelizable. Consequently, the same model runs far faster on a GPU. CPUs can still run small or quantized models and handle data prep, scheduling, and non-GPU-friendly work, but as parameter counts grow, prompts lengthen, or generation speed matters, GPUs become essential.

However, simply "having a GPU" does not guarantee good LLM performance. Four questions decide suitability:

Can the model fit in memory?

Is GPU compute fast enough?

Can weights and KV cache reach the GPU in time?

Can the inference framework fully utilize the hardware?

The article maps each hardware concept to a factory analogy:

GPU compute units → Machining tools → Execute matrix multiplies, attention, etc.

VRAM / unified memory → Warehouse & workbench → Store model weights, KV cache, temp data

Memory bandwidth → Conveyor belt → Determine how fast weights & cache reach compute units

Inference framework & driver → Factory scheduling system → Determine how much theoretical performance is realized

GPU Compute Units: "How Fast It Calculates"

LLMs consist largely of matrix math. GPUs excel because their many parallel units perform vast numbers of multiply-add operations simultaneously. The GPU is not where the model is stored — it is where computation happens. Weights are first loaded into VRAM or unified memory, then read by the GPU for forward passes.

NVIDIA cards expose CUDA Cores and Tensor Cores; AMD has its own compute units and matrix accelerators. Comparing raw core counts across vendors or architectures is meaningless without considering data precision, clock speed, dedicated matrix units, and software support.

FLOPS vs OPS

Two metrics appear in specs:

FLOPS (Floating-point Operations Per Second): Floating-point ops/sec — typical for FP32, FP16, BF16, FP8

OPS (Operations Per Second): Broader ops/sec definition — typical for INT8, INT4, other low-precision AI compute

FLOPS measures floating-point; OPS covers any operation. AI chips quote FLOPS for float precision and OPS for integer/low-precision. Always read the metric together with its precision, e.g., FP16 TFLOPS or INT8 TOPS.

T, P, E Prefixes

GPU throughput uses SI prefixes:

Tera (T) = 10^12 → TFLOPS, TOPS

Peta (P) = 10^15 → PFLOPS, POPS

Exa (E) = 10^18 → EFLOPS, EOPS

1 TFLOPS = 10^12 FLOP/s
1 TOPS   = 10^12 OP/s
1 PFLOPS = 10^15 FLOP/s
1 EFLOPS = 10^18 FLOP/s

These prefixes only indicate order of magnitude.

Why Peak Numbers Cannot Be Compared Directly

FLOPS/OPS represent theoretical peaks at a specific precision. When evaluating a number, you must confirm:

Precision: FP32, FP16, BF16, FP8, INT8, INT4?

Unit type: general cores or Tensor Cores / matrix accelerators?

Density: Dense or Sparse performance?

Framework support: does the inference engine actually invoke that compute path?

Quantized models often mix integer, FP16, and FP32 kernels, so advertised peak TFLOPS/TOPS ≠ real inference speed.

VRAM / Unified Memory Capacity: "Can It Run at All?"

Discrete GPUs use dedicated VRAM; integrated GPUs share system memory (unified memory). At runtime you must fit:

Model weights

KV cache

Framework overhead

Temporary tensors & memory fragmentation

Desktop display / other GPU processes

Rough Weight Size Estimation

Model weight size ≈ parameter count × bytes per parameter

FP16: 2 bytes per parameter

INT8: 1 byte per parameter

Q5 family: ~0.65–0.75 bytes per parameter

Q4 family: ~0.5–0.6 bytes per parameter

Q4/Q5 formats also store scaling factors, group metadata, etc., so actual file size ≠ parameter count × 4 or 5 bits.

Why Weight Size ≠ Total Memory Use

Weights are only part of the footprint. KV cache grows with context length; temporary tensors and framework overhead add more. Therefore, you cannot size VRAM based solely on model file size — reserve headroom for context, display, and runtime data.

What Happens When Capacity Is Insufficient

On discrete GPUs, frameworks may offload layers to system memory (CPU offload), computing on CPU or streaming via PCIe. The model may still run but much slower. If total memory (VRAM + RAM) is still inadequate, swapping, load failures, or OOM errors occur. Capacity answers "can it run stably", not "how fast".

Memory Bandwidth: "How Fast Data Is Fed"

Bandwidth (GB/s) describes the transfer rate between VRAM/unified memory and GPU compute units. Example: 256 GB/s means up to 256 GB/s ideally. Even with massive compute, if weights and cache cannot arrive fast enough, compute units starve.

Capacity vs Bandwidth

Memory Capacity → How much weight & runtime data fits → Warehouse size

Memory Bandwidth → How much data/sec reaches GPU → Conveyor belt speed

Large capacity solves "fit"; high bandwidth solves "feed fast". Hence large unified memory isn't necessarily faster than smaller dedicated VRAM.

Why Bandwidth Affects Generation Speed

During token-by-token generation, parallelism per request is low, so compute units often wait for data. Single-user throughput correlates strongly with effective memory bandwidth.

Peak Bandwidth ≠ Effective Bandwidth

Specs give theoretical peaks. Real effective bandwidth depends on:

Memory type, frequency, bus width, channel config

Framework & quantized kernel access patterns

KV cache size & context length

OS & other process memory contention

Power, temperature, sustained clocks

Peak bandwidth sets an upper bound; final speed must be measured with the same model, quantization, and framework.

Inference Framework & Driver: "How Much Hardware Potential Is Unlocked"

Hardware specs are the foundation, but drivers and frameworks determine realized performance.

GPU Driver → Identify hardware, provide low-level APIs → NVIDIA Driver, AMD Driver

Compute Backend → Expose GPU compute to frameworks → CUDA, ROCm, Vulkan, Metal

Inference Framework → Load model, orchestrate inference pipeline → llama.cpp, Ollama, vLLM, TensorRT-LLM

Compute Kernels → Execute matrix, quantization, attention ops → CUDA Kernel, ROCm Kernel, FlashAttention

Even with similar hardware specs, actual speed can differ significantly due to:

Support for the model's quantization format

Whether all layers can be fully offloaded to GPU

Availability of architecture-optimized kernels

Support for efficient KV cache, batching, attention optimizations

Driver, framework, OS compatibility

Thus, GPU selection must consider both hardware specs and mature software support for the target model and framework.

Four Questions Summary

Can the model fit? → Primary: VRAM / unified memory capacity → Also: Quantized size, KV cache, framework overhead

Time to first token? → Primary: Effective GPU compute → Also: Prompt length, framework & kernel efficiency

Is output streaming smooth? → Primary: Memory bandwidth → Also: Weight size, KV cache, quantized kernel

How much theoretical performance is usable? → Primary: Inference framework & driver → Also: Compute backend, kernel optimizations, compatibility

When evaluating a graphics card, ask these four questions in order to match hardware to your LLM inference needs.