Why Round‑Robin Fails for LLM Inference and How llm‑d Fixes It

LLM inference does not scale like ordinary web services; using a standard Kubernetes Service with round‑robin forwarding discards the acceleration provided by KV caches because each pod recomputes the full context for every request.

KV caching stores the Key and Value matrices produced during attention. A 13B model generates roughly 1 MiB of cache per token, so a 4 KiB context consumes about 4 GiB of GPU memory. When two requests share a prefix (e.g., the same system prompt), the second request can skip the prefilling stage if it lands on the same replica, a technique called prefix caching.

Four Kubernetes questions llm‑d addresses:

Which replica holds which prefix? Each inference server emits events when creating or evicting cache blocks; the router maintains a real‑time index and routes requests to the replica that already holds the needed prefix.

When should the index be ignored? Cache affinity can overload a hot replica. llm‑d switches between sticky routing (keep traffic on the hot replica) and pure load‑balancing (ignore affinity) once the replica reaches a saturation threshold, a policy it calls “sticky until saturated”.

Where to place the cache? GPU memory fills quickly. llm‑d evicts cache blocks to CPU memory and then to disk. On a cluster of four H100 GPUs serving 250 concurrent users, this hierarchical eviction yields a 13.9× throughput improvement compared with keeping all cache on the GPU.

Should prefilling and decoding be separated? Prefill is compute‑bound, decode is bandwidth‑bound. Splitting them into independent pools let AWS achieve up to a 70 % increase in tokens / second on GPT‑OSS, at the cost of an extra network hop for KV cache before the first token.

After applying all four solutions, the same hardware and model deliver roughly three‑fold higher output throughput and halve the first‑token latency.

What is llm‑d? llm‑d is an open‑source project that implements the above four mechanisms on top of vLLM or SGLang without replacing them. It sits between Kubernetes and the model server, using the Gateway API Inference Extension to sense each replica’s cache and load state and forward requests accordingly. The project is Apache 2.0 licensed, a CNCF sandbox project, and is backed by Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA, AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, among others. Production users include Tesla, Snowflake, Cohere, and DigitalOcean.

Official performance numbers show Wide Expert‑Parallelism achieving about 50 k tokens / s cluster throughput on a 16 × 16 B200 configuration, with single‑card decode around 3 100 tokens / s. Google’s prediction‑latency scheduling reduces TTFT and ITL by 40 %, and Oracle’s prefill/decode split raises throughput by 10‑30 % on comparable infrastructure.

Case study – Tesla & Red Hat: Tesla initially used vLLM with a Kubernetes StatefulSet and faced three problems: massive model files (hundreds of GB) slowed loading over NFS, pod‑node binding required manual PVC deletion on hardware failure, and round‑robin routing wasted GPU KV cache. After switching to KServe + llm‑d, the routing layer (via Envoy AI Gateway) performed prefix‑aware forwarding. Running Llama 3.1 70B on four × MI300X GPUs (tensor‑parallel = 4, gpu‑memory‑utilization = 0.90, max‑model‑len = 65536) increased token‑per‑second output by three times and cut TTFT in half. The performance curve after the routing change is shown below.

During the migration, Tesla’s team contributed upstream fixes to KServe: making storageInitializer optional (PR #4970) and updating to the latest Gateway API Inference Extension (PR #4886).