Avoiding Pitfalls in Heterogeneous Token Factories: Industry‑Level Design Practices for Cross‑Hardware LLM Inference

As large‑model inference moves toward cost and compute constraints, Prefill‑Decode (PD) heterogeneous inference has shifted from a research prototype to production deployment. Different accelerators excel at different stages—compute‑intensive Prefill benefits from high‑throughput chips, while bandwidth‑intensive Decode requires hardware with strong memory bandwidth and KV cache capacity.

The paper "Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving" (arXiv:2606.29708) jointly authored by several Shanghai research institutes systematically dissects this design space, extracting three core boundary decisions and nine practical deployment guidelines, and validates them on a MuXi C600 GPU + NVIDIA Hopper GPU production cluster.

Core question: When heterogeneous inference combines different accelerators, numeric formats, interconnect paths, and KV‑cache residency layers, which design decisions must be made jointly at the PD boundary and which can be made independently?

The authors introduce five design axes—accelerator selection, precision format, interconnect, KV‑cache placement, and runtime environment—and a key abstraction called Runtime KV State, which encapsulates the KV tensor, its format, metadata, residency, and ownership.

Three Mandatory Boundary Decisions

Decision 1: Compute Placement – Choose which accelerator pool handles Prefill and which handles Decode. The choice is not merely “fastest hardware for each stage”; precision support is tightly coupled to the accelerator, so compute placement, precision selection, and load balancing must be decided together.

Decision 2: KV Representation – Define how Runtime KV State is represented, transmitted, and consumed across the PD boundary. Existing KV‑cache transfer engines (e.g., NIXL, Mooncake) move raw bytes without preserving tensor semantics, leading to silent semantic failures when producer and consumer interpret formats differently.

KV portability is defined as Decode’s ability to consume the transferred state directly or after an explicit, verified conversion. Incompatible invariants (model identity, adapter, token range) and convertible differences (layout, partition, numeric representation) must be distinguished.

Decision 3: KV Ownership & Lifecycle – Manage the full lifecycle of Runtime KV State after transmission: capacity reservation, ownership transfer, resource release, and failure handling. Source‑level analysis of vLLM (NIXL pull mode) versus SGLang (proactive allocation) shows divergent capacity accounting, fault recovery, and congestion‑control behaviors.

Nine Deployment Best Practices

Through production‑cluster measurements, source‑code audits of vLLM/NIXL and SGLang/Mooncake, and controlled single‑node experiments, the paper derives nine actionable rules covering hardware selection, quantization configuration, KV‑cache transmission, and full‑lifecycle management.

Production Case: CPHD‑GLM5.1

The authors implement a heterogeneous token factory named CPHD‑GLM5.1: Prefill runs on MuXi C600 GPU with INT8/W8A8 precision, Decode runs on NVIDIA Hopper GPU with FP8, serving the GLM‑5.1 model. At 64K input length and 90 % prefix cache hit rate, key metrics remain within SLA, and benchmark results (AIME 25, AIME 26, SWE‑Bench Verified) show negligible quality deviation.

Controlled Experiments Reveal Coupling Effects

Single‑node SLA tests on Qwen3‑32B, SGLang PD, and NIXL demonstrate that compute placement and KV format are inseparable: switching from a 6P2D to a 4P4D topology halves the maximum request injection rate, but changing KV representation from BF16 to FP8 e4m3 restores the rate to the original level, proving that KV dtype directly influences Decode bandwidth and tail latency.

Further experiments show asymmetric latency impacts of precision strategies: FP8 improves total‑time‑to‑first‑token (TPOT), while AWQ INT4 boosts throughput‑to‑first‑token (TTFT). These results reinforce that precision choices belong to the runtime role rather than a global setting.

Open Challenges

Cross‑vendor hardware KV unified transmission stack – existing adapters only wrap vendor‑specific communication libraries, lacking a native, standardized cross‑hardware abstraction.

Co‑planning of interconnect network and PD resources – current deployments treat network bandwidth as fixed; an integrated methodology to jointly design topology, bandwidth, and PD hardware pools is missing.

The article concludes that a systematic design‑space map and the identified best practices provide a practical foundation for engineers building scalable heterogeneous LLM inference services.