How Continuous Batching Boosts LLM Throughput by Up to 36×

Problems of Static Batching

Static batching groups multiple requests into a single batch that runs from start to finish before accepting new requests. When request lengths differ—e.g., request 1 generates 10 tokens while request 8 generates 500—the early‑finishing request occupies a GPU slot for the remaining 490 iterations, creating the straggler problem. GPU utilization often falls to 20%–30% because the slowest request dictates the pace.

Continuous Batching: Iteration‑level Scheduling

Continuous batching reevaluates the entire batch at each forward‑propagation step instead of locking the batch until all requests finish. When a request ends, its slot is released immediately, and the scheduler can insert a new request in the next iteration if there is enough free KV‑cache memory. This keeps the batch composition dynamic and the GPU fully occupied.

For example, with 8 GPU slots and 20 queued requests, static batching would keep a slot idle after request 1 finishes at iteration 10 until request 8 completes at iteration 500. Continuous batching would fill the freed slot at iteration 11, eliminating idle time.

Relation to KV Cache

Each request in continuous batching needs its own KV‑cache block, which grows with each generated token. Without PagedAttention, the system would have to pre‑allocate a large contiguous memory block for the worst‑case output length, leading to either massive waste or crashes.

PagedAttention allocates fixed‑size small KV blocks on demand, avoiding the need for contiguous memory. When a request finishes, its blocks are released instantly for new requests. Continuous batching manages the timing of request entry/exit, while PagedAttention manages the memory layout; both are required for full effectiveness.

Throughput vs. Latency

Increasing batch size raises overall token throughput but also increases per‑request latency because each request competes for GPU cycles. A batch size of 2 gives low latency but leaves many slots idle, reducing throughput. Conversely, a large batch maximizes GPU utilization but can make individual responses slower. Production systems typically find a sweet spot empirically.

Chatbot services that serve thousands of users prioritize throughput, accepting higher latency, while latency‑sensitive tasks like real‑time code completion may sacrifice some throughput for faster responses. Many systems expose batch‑size as a configurable parameter for users to balance these trade‑offs.

Application Scenarios

Continuous batching is now standard in mature inference stacks: vLLM, TensorRT‑LLM, and SGLang all implement it. The idea originated from the 2022 Orca paper, which reported up to a 36× throughput increase by switching from static to continuous batching. Without it, production LLM services often achieve only 20%–30% GPU utilization.

Conclusion

Continuous batching repeatedly reassesses the batch each iteration, instantly replacing completed requests, and keeps the GPU fully loaded. Combined with PagedAttention, it resolves memory‑allocation challenges and delivers substantial throughput gains, fundamentally changing how modern LLM inference systems are built.