CloudMatrix384: Co-Designing Supernodes for Trillion-Parameter LLM Inference

Hardware Topology: All-to-All Peer Interconnect and Three-Plane Network

CloudMatrix384 integrates 384 Ascend 910 NPUs and 192 Kunpeng CPUs via a Unified Bus (UB) network that enables full peer-to-peer connectivity. Unlike traditional hierarchical designs with high intra-node but low inter-node bandwidth, the UB network keeps inter-node bandwidth decay within 3% and latency increase under 1 µs .

The system provides three physically separate network planes:

UB Plane : Primary intra-supernode data path carrying tensor parallelism, expert parallelism, and NPU remote DRAM access requests.

RDMA Plane : Cross-supernode channel for KV Cache migration between prefill and decode stages, and for multi-supernode cluster deployment.

VPC Plane : Management control plane, persistent storage access, and external service communication.

The RDMA/VPC separation ensures compatibility with existing data centers; the long-term direction is to unify them into a single UBoE plane to build availability-zone-level AI clusters with the supernode as the base unit.

Each Ascend 910 uses a dual-die package: 24 AI Cube cores (matrix) + 48 AI Vector cores (vector) per die. The AIV-Direct mechanism lets AIV cores write directly into remote NPU on-chip memory via UB, bypassing SDMA engine launch overhead — a hardware foundation for later MoE communication optimizations.

Service Architecture: PDC Three-Plane Decoupling and Peer Scheduling

CloudMatrix-Infer decomposes inference into three independent subsystems — Prefill , Decode , and Caching — operating as peer resource pools rather than around a centralized scheduler. This contrasts with Mooncake and NVIDIA Dynamo , where schedulers must balance load against KV Cache locality because cross-node cache access is expensive. In CloudMatrix-Infer, KV Cache resides in a distributed memory pool accessible uniformly via UB; all NPUs see identical bandwidth and latency. The scheduler only picks the least-loaded instance, dramatically simplifying logic and enabling direct elastic scaling.

The cache subsystem is backed by the Elastic Memory Service (EMS) , which provides two layers: context cache for historical KV Cache reuse and model cache for accelerated model switching. EMS builds a distributed memory pool on CPU-side DRAM and uses UB DMA for zero-copy NPU access to remote DRAM, delivering far higher bandwidth than the traditional VPC path.

Decode Phase: System-Level Optimization for Massive Expert Parallelism

DeepSeek-R1 decode deploys EP320 expert parallelism — 320 NPU dies each host one expert (256 routed, 32 shared, 32 redundant for load balancing). With a local batch of 96, the global batch reaches 30,720 , making communication overhead the primary bottleneck.

FusedDispatch and FusedCombine

Traditional All-to-All requires three data exchanges: routing metadata, token data, and expert output return. CloudMatrix-Infer fuses these into two custom operators with four key decisions:

AIV-Direct Write : Sender AIV cores write directly into pre-allocated on-chip buffers on receivers, avoiding SDMA launch latency.

Early INT8 Quantization : Token data is quantized to INT8 before transmission, reducing per-token communication volume from 14 KB (BF16) to 7.5 KB (including scale factor) .

Double-Buffered Static Pre-allocation : Separate buffers for Dispatch and Combine (≈ 645 MB/die total) eliminate dynamic allocation and race conditions, enabling static graph execution.

Token-Level Pipeline : Load, quantize/address, and remote write stages are pipelined at single-token granularity, overlapping compute and communication.

Measured at EP256 , CANN EP achieves lower latency than DeepEP on H800 : Dispatch 152 µs vs 194 µs ; Combine 149 µs vs 360 µs .

MLA Operator Fusion and NZ-Format KV Cache

DeepSeek's Multi-head Latent Attention (MLA) involves many fine-grained ops (RMSNorm, linear projections, RoPE). CloudMatrix-Infer fuses pre-attention ops into MLAProlog and attention core plus data reshaping into Fused Attention , collapsing multiple kernel launches into two . KV Cache is stored in NZ format — a hybrid layout matching Cube core matrix multiplies — with format conversion done on write, avoiding ND-to-NZ conversion at inference time and saving memory bandwidth.

Micro-Batch Decode Pipeline

The decode pipeline splits the Attention path and MoE path into two streams with asymmetric AIC/AIV allocation: Attention gets 16 AIC + 32 AIV , MoE gets 8 AIC + 16 AIV . Both paths hit ~ 600 µs latency under typical load, enabling perfect overlap of two micro-batches. This reduces per-layer latency by ~ 10% and boosts throughput 6–9% .

Pipelined Multi-Token Prediction (MTP)

Naive MTP requires k+1 sequential graph launches, each incurring 0.6–0.8 ms CPU-NPU sync. CloudMatrix-Infer precomputes all metadata (sequence lengths per stage) at step start, writes them to NPU memory, and moves sampling entirely onto the NPU fused into the graph. The NPU then executes continuously without CPU intervention.

Prefill Phase: Hybrid Parallelism and Heterogeneous Pipeline

The core prefill challenge is load imbalance from variable request lengths. DeepSeek's native pure data parallelism (DP32) leaves short requests idle while long requests stall the global barrier. CloudMatrix-Infer splits MLA computation into three stages with distinct parallelism strategies:

Stage 1 (down_proj) & Stage 3 (o_proj) : Sequence Parallelism — tokens from multiple requests are concatenated and evenly partitioned across NPUs, eliminating length disparity.

Stage 2 (Q/K/V projection + FlashAttention) : Tensor Parallelism — attention heads evenly distributed across NPUs.

Inter-stage data redistribution uses All-Gather and All-to-All; UB's high bandwidth keeps this overhead in check. The prefill micro-batch pipeline exploits Ascend's heterogeneous compute units: AIC runs heavy Attention/MLP, AIV handles token reordering and metadata generation before Dispatch, and SDMA drives All-to-All data movement. The three form a pipeline across two micro-batches, lifting prefill throughput 23–31% .

Distributed Cache: EMS Memory Pooling and Two-Tier Cache

EMS leverages UB to turn CPU-side DRAM into a globally shared memory pool with a unified key-value interface. Design highlights:

Consistent Hashing Index : Centralized controller maintains a DHT view; MP Servers manage local shards.

NUMA-Aware Huge Page Management : Supports variable-length allocation to reduce fragmentation.

DRAM-SSD Two-Tier Storage : DRAM serves as hot cache, SSD (EVS) as persistence layer. Data is written to EVS first for durability; DRAM evicts via LRU. Because data is sharded across 48 nodes , even SSD-resident data can be pulled concurrently from multiple nodes, yielding aggregate bandwidth of 48 × 400 Gbps .

Namespace Isolation : Logical isolation and capacity limits for multi-tenant scenarios.

Model caching shows dramatic gains for the 671B model: 8 concurrent instances loading from OBS took 2,560 seconds (OBS single-bucket 2.5 GB/s contention), reduced to 320 seconds via EMS shared loading, while DRAM footprint dropped from 8× model size to 1× .

INT8 Quantization: Accuracy Preservation Without Retraining

Ascend 910 lacks FP8 support while DeepSeek trains in FP8. CloudMatrix-Infer's INT8 scheme matches official API accuracy across 16 benchmarks without retraining, using:

Mixed Precision : Large matmuls in FFN/Attention quantized to INT8; sensitive layers (Norm, Gating) kept in BF16.

Adaptive Scale Search : Per-tensor offline search for optimal scaling factors minimizing quantization error.

Outlier Suppression : Linear transforms redistribute range for long-tail components like MoE gating.

Block-Wise Clipping with Error Compensation : Large weight tensors split into blocks, each with independent clipping range and a lightweight compensation term.

Mixed-Granularity Quantization : Activations per-token (dynamic), weights per-channel (static), matching Ascend's integer matmul instructions.

Measured Performance (256 NPUs, DeepSeek-R1 671B INT8)

Prefill : 6,688 tokens/s/NPU , compute efficiency 4.45 tokens/s/TFLOPS — exceeds SGLang on H100 (3.75) and DeepSeek profile on H800 (3.96) .

Decode : TPOT 49.4 ms , 1,943 tokens/s/NPU , compute efficiency 1.29 tokens/s/TFLOPS — leads baselines. Under strict 15 ms TPOT constraint (batch=8), sustains 538 tokens/s/NPU .

Context Cache : At 90% hit rate , prefill throughput 2.28× , TTFT -59% . UB plane delivers up to 1.52× throughput advantage over VPC plane for cache access.

Communication microbenchmarks at EP256 show CANN EP Dispatch/Combine latency both lower than DeepEP on H800; Combine bandwidth reaches 103 GB/s vs H800's 40 GB/s , directly attributable to UB's low latency and AIV-Direct.

Future Evolution Directions

Larger Supernodes : Production-trace simulations show scaling beyond 384 nodes significantly improves resource allocation rate (avg block size 10.08: 384 nodes → 94% vs 224 nodes → 91%) with near-flat network cost when switch ports are fully utilized.

CPU Physical Decoupling : Current fixed CPU:NPU ratio on same board; future separates them into independent node types for independent scaling of NPU compute density and CPU memory capacity.

Component-Level Decoupling : Split Attention and MoE into independent microservices, each deployed on best-fit hardware with independent elastic scaling. The Adrenaline system has demonstrated offloading decode-side Attention to idle prefill instances.

Network Plane Unification : Merge RDMA and VPC into a single UBoE plane, building availability-zone-level AI clusters with the supernode as the fundamental unit.

Conclusion

CloudMatrix384 and CloudMatrix-Infer illustrate a recurring theme: when both model and hardware scale, the software bottleneck is rarely raw operator speed but rather whether scheduling is bound by data placement, whether communication overlaps compute, and whether resources flow fluidly across stages . Decoupling — prefill from decode, cache from compute, Attention from MoE, CPU from NPU — runs through the entire design. The coordination complexity this introduces is outweighed at supernode scale by gains in throughput, latency, and resource utilization.