Li Xiuhong on Cross-Cluster Heterogeneous PD Separation and Token Factory “Super Pipeline” at WAIC

In the era of agents, inference workloads become multiplicative rather than additive, while compute growth remains linear, creating a structural supply‑demand gap. 无问芯穹 addresses this with the “front‑store, back‑factory” Agentic Infra strategy announced at WAIC 2026, which centers on two anchors—scale and efficiency—and three core components: a compute‑aggregation center, a Token Factory (Agentic MaaS), and an AI productivity store.

The Token Factory introduces the cross‑cluster heterogeneous PD‑separation architecture (Prefill‑RelayDecode‑MainDecode, PDD). Real‑world tests show PDD reduces first‑token latency (TTFT) by 51.5% and per‑token cost by 37.5% compared with a baseline that keeps Prefill and Decode in the same cluster.

PD separation splits the two inference phases: Prefill, which is compute‑intensive, and Decode, which is memory‑intensive. Different chips excel at different phases, so heterogeneous deployment lets each chip do what it does best. The architecture adds two orthogonal dimensions—heterogeneity (different chips or configurations) and cross‑cluster placement (different data‑centers).

Cross‑cluster KV‑Cache transfer is the primary bottleneck. A 64 KB request with 90 % prefix hit rate generates ~50 KB KV‑Cache per token; a 100 K‑token request can reach gigabytes of state, far exceeding typical inter‑data‑center Ethernet bandwidth (10‑100 Gbps). The team’s feasibility analysis shows that a 90 % hit rate cuts transferred data by an order of magnitude, making the bandwidth requirement comparable to the compute capacity.

To hide the remaining latency, the system adds a Relay‑Decode (RLD) stage in the same rack as Prefill, which immediately starts decoding using the locally transferred KV‑Cache via RDMA. While the remote Main‑Decode (MD) waits for the KV‑Cache over Ethernet, RLD emits “eager output tokens” to the user, masking the long‑haul delay.

The final handoff from RLD to MD sends only token IDs (≈1.5 KB for 100 tokens) instead of the full KV‑Cache. MD reconstructs the missing KV‑Cache locally, leveraging the memory‑intensive nature of Decode to absorb the extra compute cost. Micro‑benchmarks show this approach reduces transmission load from 4.6 MB to 1.5 KB per 100‑token segment and caps latency at ~321 ms with minimal jitter, compared with up‑to‑512 ms and additional GPU copy overhead for full KV‑Cache transfer.

Two handoff modes are offered: One‑Shot (single large transfer) and Catch‑Up (multiple small transfers). The system can switch dynamically based on RLD load, balancing latency spikes against extra RLD work.

Cost analysis using the Benefit‑Cost Ratio (BCR) indicates PDD achieves up to 37.5 % higher BCR than a same‑rack homogeneous baseline, while using fewer GPUs (43 A + 32 B vs. 88 A) and a modest cross‑rack link, yielding a total hardware cost reduction of 3.5‑7.1 % and a 27.8 % throughput increase. The RLD stage consumes only ~6 % of total token generation resources, demonstrating that a tiny “relay” component can unlock large cross‑cluster gains.

Long‑term observations reveal a non‑monotonic relationship between KV‑Cache hit rate and profitability: beyond a certain “bathtub” region (≈20‑40‑70 % hit), higher hit rates can actually reduce net profit because token pricing discounts outweigh throughput gains. The team mitigates this by filtering “anomalous traffic” (≈3‑4 % of requests) that would otherwise dominate bandwidth.

Looking ahead, the interview concludes that cross‑cluster heterogeneous inference will become standard as cluster counts grow from tens to thousands and chip diversity expands. PDD’s core insight—re‑ordering computation to outrun transmission limits—offers a scalable path for AI infrastructure to keep pace with exponential demand while keeping costs down.