Advanced AI Infra: Making Large Language Models Produce Deterministic Outputs

Background

When interacting with large language models (LLMs), the same prompt can produce different outputs each time because the model behaves like a stochastic sampler. Reproducibility, however, is essential for scientific progress and for reinforcement‑learning‑based rollouts that require deterministic behavior.

Why Floating‑Point Addition Is Not Associative

Floating‑point numbers follow the IEEE‑754 standard, where addition does not satisfy the associativity law (a + b) + c ≠ a + (b + c). The article demonstrates this with a concrete example: adding 2048 and 1 on FP16 hardware yields 2049 internally, but when the result is rounded back to the nearest even value it becomes 2048 again.

Batch Invariance vs. Run‑to‑Run Invariance

Two kinds of determinism are distinguished:

Run‑to‑run invariance : With a fixed batch size and no concurrent requests, the same prompt always yields the same answer.

Batch invariance : The output must stay identical even when the batch size or the number of concurrent requests changes.

Batch variance originates from dynamic scheduling decisions that alter the reduction tree of floating‑point additions.

GEMM Tiling and Split‑K

Matrix multiplication (GEMM) is the core of transformer inference. Modern GPUs tile the operation to fit into shared memory (SMEM) and registers. The tiling parameters BLOCK_M, BLOCK_N, and especially BLOCK_K determine how the reduction over the K dimension is performed. Changing BLOCK_K reshapes the reduction tree and therefore changes the addition order.

When the K dimension is very large but the spatial dimensions M and N are small, the scheduler may split the K dimension across multiple thread blocks ( Split‑K ). Each block computes a partial result and the final answer is obtained with an atomic_add or a workspace reduction. Because the order in which the partial results are summed depends on the runtime scheduling, Split‑K introduces batch variance.

vLLM’s Deterministic GEMM Strategies

vLLM enforces batch invariance by fixing the tiling configuration:

On Ampere (SM80) GPUs it disables Split‑K and locks BLOCK_K to a constant value.

On Hopper/Blackwell (SM90/SM100) it only disables Split‑K, because the newer hardware already uses a deterministic reduction topology.

Both cases fall back to a handcrafted Triton kernel called matmul_persistent that uses a static configuration (e.g., BLOCK_K = 64) and never invokes autotuning or atomic reductions.

RMSNorm Batch Invariance

RMSNorm normalizes each token independently. To keep the reduction order stable, vLLM forces a fixed block size (1024 threads) for the Triton kernel _rms_norm_kernel. When the number of tokens is small, the kernel still uses the same block size, ensuring that the tree‑reduction pattern does not change with batch size.

Attention and FlashAttention

FlashAttention computes the Q‑K dot product, applies scaling, masking, softmax, and finally multiplies by V. The kernel tiles the query dimension (BLOCK_M) and the key/value dimension (BLOCK_N). Batch variance appears when the KV dimension is split across multiple thread blocks ( Split‑KV ) during decoding. vLLM eliminates this by forcing num_splits = 1, which makes the reduction tree identical for every batch size.

NCCL All‑Reduce and Distributed Determinism

In tensor‑parallel inference, each GPU computes a partial result that must be summed across devices with an NCCL All‑Reduce. NCCL chooses the reduction algorithm (Ring, Tree, Simple, etc.) and the number of communication channels based on message size. Because the message size depends on the batch, NCCL may switch algorithms, causing a different addition order.

vLLM’s batch‑invariant mode disables custom all‑reduce paths, forces a single NCCL channel, selects the tree algorithm with the Simple protocol, and disables CollNet, NVLS and other optimizations. This fixes the reduction topology regardless of batch size.

Summary

The root cause of batch variance in LLM inference is the dynamic alteration of reduction topologies in GEMM, RMSNorm, FlashAttention, and NCCL All‑Reduce. By fixing tiling parameters, disabling Split‑K/KV, and locking NCCL communication settings, vLLM achieves true batch invariance at the cost of some throughput. The article provides a step‑by‑step walkthrough of the underlying hardware mechanisms, the software heuristics, and the concrete code paths used to enforce determinism.