Why Your GPU Idles During LLM Inference: Static vs Dynamic vs Continuous Batching

Why Batching Exists

Modern GPUs like the A100 deliver ~312 TFLOPS of BF16 compute but only ~2 TB/s memory bandwidth. Decoding is entirely memory-bound: reading model weights dominates latency, leaving compute units idle. Batching amortizes the weight-read cost across multiple sequences — weights are loaded once and reused for all sequences in a forward pass.

Throughput rises sharply with batch size until a knee point where the workload shifts from memory-bound to compute-bound. Below the knee, batching is nearly free; above it, each additional sequence costs compute time. The knee varies by model, GPU, and sequence length, so measure on your hardware.

What Breaks for LLMs

Traditional batching works for classifiers/embeddings because: one forward pass yields the full answer, rows are independent, and per-row cost is known upfront. Autoregressive LLMs break all three:

One forward pass produces one token, not the answer (e.g., 400 tokens = 400 passes).

Rows depend on their own history via KV cache written each pass.

Cost is unknown until the model emits a stop token (could be 12 or 4000 tokens).

Padding only equalizes input width, not the time a request occupies the GPU. A fixed batch runs at the speed of its slowest member.

Static Batching

Collect a fixed set of requests, run together, return all when the last finishes. Fast finishers wait for the slowest, wasting GPU slots and increasing user latency. Waste grows with output-length variance. Anyscale measured OPT-13B on 40GB A100: increasing output-length disparity dropped static batching to ~81 tokens/sec, while continuous batching maintained an order of magnitude higher throughput.

Static batching remains suitable for fixed-output tasks (classification, embedding, scoring) where every request finishes almost simultaneously.

In vLLM, configure static batching by passing the entire workload to llm.generate at once, setting a max_tokens ceiling, and keeping prompt sizes similar so each sequence finishes at nearly the same time.

Dynamic Batching

Adds a timer: batch fires when size limit reached or window expires. Reduces queue wait for early arrivals but still holds the batch until all members finish. The slow-member problem persists. Dynamic batching makes one decision per batch then hands off to the engine — fine for fixed-output models (e.g., Triton).

Continuous Batching

Scheduler runs one iteration, reclaims control, and decides again. When a sequence emits its final token it leaves the batch; waiting requests fill the slot next iteration. Batch composition changes every iteration — iteration-level scheduling. No slot waits for the slowest sequence, keeping GPU saturated even with highly variable output lengths.

Limitation is memory, not compute. Each active sequence holds a growing KV cache; the cache pool (not math) determines how many sequences fit. On a 40GB A100 with a 13B model, only a few long sequences fit. When cache pool exhausts, the scheduler evicts a running request and recomputes its prefill later — effectively doing the same prefill twice.

Chunked Prefill

Continuous batching introduces stalls: a new request with a long prompt (e.g., 32K tokens) monopolizes an iteration for prefill, blocking all active decoding. Chunked prefill splits the prompt into fixed-size token chunks (e.g., 2K tokens) scheduled across multiple iterations, extending the KV cache incrementally. Attention remains correct because later chunks attend to earlier chunks' KV entries. First token arrives after the last chunk.

Prefill is compute-intensive, decode is memory-intensive; mixing both in a batch utilizes both parts of the chip.

Smaller chunks give scheduler more decode opportunities, reducing inter-token latency (ITL) spikes.

Larger chunks process new prompts more efficiently, usually improving time-to-first-token (TTFT), but active decoders wait longer between tokens.

Too-small chunks may lower GPU utilization and increase attention overhead because later chunks must re-read KV entries created by earlier chunks.

Chunked prefill is enabled by default in vLLM V1 via --max-num-batched-tokens limiting tokens per iteration. SGLang uses --chunked-prefill-size (-1 disables).

Conclusion

The three strategies differ only in when they fix the batch, each suiting different workload shapes. For real-time traffic with variable output lengths, continuous batching is the only strategy that sustains performance, and every mainstream engine now provides it by default.