Advanced AI Infra: Making Large Language Models Produce Deterministic Outputs
This article analyzes why LLM inference often yields nondeterministic results, explains how floating‑point addition order, GEMM tiling, Split‑K, RMSNorm, FlashAttention, and NCCL all contribute to batch variance, and details the engineering steps vLLM takes to enforce batch‑invariant execution across GPUs.
