xDeepServe on CloudMatrix384: Full-Stack Design for Large-Scale MoE Model Serving

The article analyzes Huawei's xDeepServe, a production-grade LLM serving system built for the CloudMatrix384 supernode that interconnects 384 Ascend 910C chips via a high-speed UB network providing a globally shared memory space with low-latency unified access. As MoE models grow, their fine-grained routing, synchronization, and load-balancing needs clash with traditional homogeneous, static-parallel serving designs; xDeepServe addresses this gap through full-stack redesign across communication primitives, service engine, execution architecture, and reliability.

1. XCCL Communication Library: Microsecond Primitives on Global Shared Memory

CloudMatrix384's UB network exposes a globally addressable space across 384 NPUs with DMA and memory semantics. XCCL exploits this to provide three key primitives:

Point-to-Point Transfer

Used for KV Cache migration between prefill and decode. Each NPU's on-chip memory is split into application, metadata, and managed regions. Sender uses MTE2 to move data from application region to AIV core's Unified Buffer, then MTE3 writes into receiver's managed region while updating receiver metadata's tailPtr. Receiver polls, pulls data to local application region, and acknowledges. The protocol bypasses CPU entirely; end-to-end latency stays under 20µs for payloads ≤1MB.

All-Gather (Dispatch/Combine)

In expert parallelism (EP), each token routes to Top-k experts and outputs are weighted-aggregated. XCCL's Dispatch/Combine uses a pull model: sender writes per-expert token shards into managed regions and updates per-receiver metadata (token counts, offsets). All NPUs concurrently pull once metadata from all peers arrives. At EP128 with per-die batch 96 (global batch 12288), Dispatch latency remains low; with INT8 quantization, Dispatch latency even drops below Combine as data volume grows.

Asymmetric Routing (A2E/E2A)

When MoE and Attention NPU counts differ (e.g., 288 vs 160), a pure pull model forces Attention NPUs to push metadata to many expert NPUs, bottlenecking AIV scalar throughput. XCCL introduces a "trampoline forwarding" mechanism: a subset of expert NPUs equal to Attention NPU count act as first-hop relays. Attention data goes to relays first, then relays forward to remaining experts. This two-stage routing cuts total metadata update overhead; at global batch 46080, A2E and E2A latencies are held to 172µs and 193µs respectively.

2. FlowServe Service Engine: Decentralized Supernode Scaling

FlowServe centers on the Data Parallel Group (DP Group) abstraction. Each DP group is a self-contained serving pipeline (tokenization, API parsing, SPMD execution, RTC cache, network stack). Scheduling, output handling, expert load balancing run independently inside each DP group; only request dispatch, global expert rebalancing, and health checks use a centralized TE-Shell for coordination.

Request-Level Load Balancing

Prefill uses a single-level cooperative scheduler: DP-0's leader gathers DP states via All-Gather and assigns request batches using a cost model that includes prefix-cache hit rate, avoiding two-level scheduling tail latency. Decode first drops DPs at batch capacity, then picks the one with lowest KV Cache usage (reserving space for long outputs), driven by real-time pending-request and cache stats tracked by TE-Shell.

Proactive GC and Jitter Reduction

Scaling to hundreds of DP groups caused graph-launch jitter >100ms at the first MoE Dispatch operator. Three mitigations: bind executors to dedicated CPU cores, cache PTA-compiled graphs to skip runtime guard checks, manually trigger Python GC in critical operator gaps.

Expert Load Balancing (EPLB)

At EP288 (256 routed + 32 shared experts), token distribution is heavily skewed. FlowServe uses data-driven replication: periodically collect per-expert token counts, identify hot experts, create replicas in spare slots using "least-loaded-first" placement, and hot-swap via async weight loading. At inference, a Gather operator maps logical expert IDs to physical replicas, round-robin assigning by token position in batch — no cross-NPU communication needed. This cuts forward latency by over 40%.

Multi-Token Prediction (MTP) Integration

DeepSeek models include MTP modules for speculative decoding. FlowServe pipelines MTP with main model in decode as a tight five-stage flow: MTP forward generates k candidates → sample → main model verify → resample → final accept decision. This eliminates CPU wait bubbles. The second MTP layer was fine-tuned on 28,000 internal samples, raising tokens-per-step from 2.26 to 2.35, a 9% gain over direct weight reuse.

INT8 Quantization Adaptation

Ascend 910C lacks native FP8; DeepSeek trains in FP8. FlowServe combines SmoothQuant and GPTQ post-training quantization with per-module granularity: token-wise activation quantization (one scale per token), channel-wise weight quantization (one scale per output channel). During MoE Dispatch, quantization fuses with communication operators to further reduce overhead.

3. Transformerless Execution Architecture: Toward Full Decoupling

xDeepServe splits Transformer inference into modular Attention, FFN, MoE units running on separate NPU sets, interacting via high-speed interconnect. Three evolutionary phases:

Phase 1: Prefill-Decode Decoupling

Prefill is compute-intensive with dynamic shapes → Eager mode, TP=4 sequence parallel. Decode is memory-bound with fixed shapes → Graph mode, TP=1. Different DP grouping strategies apply. Requests routed by Job Executor based on cache hit rate, load, sequence length. After prefill, only KV Cache metadata (not data) registers in RTC-DistFlow; actual data moves via XCCL P2P only after decode side confirms capacity. Prefill can run on cheaper Ascend 910B (RoCE), decode stays on 910C for UB bandwidth.

Phase 2: MoE-Attention Decoupling

At larger scale, MoE experts and Attention compute fully separate. For DeepSeek-V3/R1 on a full SuperPod (768 dies): 288 dies run EP288, 480 dies run Attention. Attention DP groups execute full MLA (Prologue, Attention, gating, output projection, A2E/E2A); MoE NPUs only run expert compute and surrounding communication. Three key techniques keep MoE NPUs saturated:

DP Domain : 480 Attention dies split into 3 DP Domains (160 DP groups each). Only one Domain communicates with MoE NPUs at a time (A2E/E2A); other Domains overlap compute/communication via micro-batching (2 micro-batches per Domain, 96 each).

Persistent Kernels : MoE NPUs run three concurrent streams (recv, compute, send) as persistent kernels busy-polling, never yielding to CPU, avoiding µs-kernel interruption by ms-scale CPU scheduling.

Trampoline Forwarding : Same as XCCL section.

Under this config (global batch 46080), per-chip throughput holds at 2400 tokens/s, TPOT ~50ms.

Phase 3 Vision: Async Dataflow Serving

Current MoE-Attention decoupling still relies on two synchronous barriers (A2E, E2A). Future aim: eliminate global sync, let tensors flow asynchronously between modules like classic dataflow architectures. Requires latency-tolerant communication protocols, event-driven schedulers, and semantic correctness under partial results.

4. Reliability Design

At supernode scale, single-device failures amplify rapidly through MoE collective communication. xDeepServe builds layered detection and recovery.

Detection

Two-level heartbeats: control plane → TE-Shell → DP group Masters. Masters respond only when event loop is alive; a stuck executor makes its Master unresponsive, accurately flagging the hang. For silent stalls in async KV transfer paths, link probing injects dummy data to distinguish decode-side saturation from link-level faults.

Recovery Evolution

Full Restart : Early small-scale — restart entire engine on failure, prioritize decode instance recovery.

P/D Independent Failover : In shared clusters, prefill and decode recover independently. Single die failure in decode triggers EPLB to dynamically shrink DP group count and EP rank, continuing in degraded mode with fewer NPUs.

Fine-Grained Fault Tolerance : Transient network faults — all DP groups roll back to previous iteration and re-execute, avoiding engine restart. On-chip memory faults — collaborate with CANN runtime to remap virtual addresses and fence bad regions; only partial KV Cache lost, causing individual request failures but no system-wide interruption.

Performance Summary

Peak decode-only config (DP288, EP288, local batch 60, global batch 17280, MTP accept rate 90%): per-chip (2 dies) throughput 2400 tokens/s, TPOT ~50ms, aggregate 345K tokens/s. Production mixed workload (input 0-64K, avg 13K; output avg 2.1K): TTFT ~900ms, avg TPOT 34.8ms, meeting 35ms SLA.

Conclusion

xDeepServe demonstrates how full-stack co-design enables efficient serving of massive MoE models on supernode infrastructure. Core contributions: 3D stacking of expensive 2nm compute chiplets with mature 5nm cache chiplets (Monaka), and end-to-end reconstruction around "decoupling" and "decentralization" from communication primitives through execution architecture. Production experience shows that when MoE models meet supernode hardware, system software must break out of traditional batch scheduling and static parallelism frameworks, moving toward modular, dataflow-oriented, asynchronous paradigms.