scaleFabric: IBGDA-Powered GPU-Autonomous Communication Validated at 100K-GPU Scale

Background: Communication Becomes the Bottleneck in Massive GPU Clusters

As GPU compute power surges, simply adding more GPUs no longer guarantees faster training. In 10K-GPU clusters, network communication already consumes 30–50% of total training time. Any link jitter or data-movement delay leaves thousands of GPUs idling. The token lifecycle — from creation to delivery — is a tight collaboration among compute, network, and storage; the weakest link dictates overall cluster efficiency.

The Traditional Path and Its "CPU Penalty"

In conventional RDMA flows, data must be copied from GPU memory to CPU memory, sent by the CPU through the NIC, and reversed on the receiver side. The CPU handles both communication instructions and work-queue management. This overhead becomes critical in Mixture-of-Experts (MoE) models where All-to-All exchanges involve huge numbers of tiny packets: CPU time to process a small-packet instruction far exceeds the actual data-transfer time , creating an asymmetric "CPU penalty."

IBGDA: GPUDirect RDMA's "Second Evolution"

GPUDirect RDMA (GDR) let NICs read/write GPU memory directly, bypassing CPU memory. However, GDR still requires the CPU to initiate communication operations. IBGDA (InfiniBand GPUDirect Async) goes further: GPU kernels directly drive and control the NIC , enabling the GPU to initiate and manage inter-node communication entirely on its own. The CPU is removed from the critical path.

GPUDirect RDMA (GDR) : NIC directly reads/writes GPU memory, bypassing CPU memory copy.

IBGDA (InfiniBand GPUDirect Async) : On top of GDR, lets GPU kernels control the NIC, bypassing CPU instruction handoff. In MoE All-to-All, eliminates the small-packet "CPU penalty" — latency reduced by two orders of magnitude. scaleFabric is China's first product to achieve large-scale IBGDA deployment.

Analogy: With GDR, the GPU can "move boxes itself" but still needs the CPU to tell it what, where, and when. With IBGDA, the GPU becomes the director — it knows what data it needs and when to communicate.

scaleFabric: First Large-Scale IBGDA Product, Validated at 100K GPUs

scaleFabric is not a lab prototype; it has been deployed in a 100K-GPU production cluster. Key measured metrics:

Forwarding latency: 260 ns

End-to-end latency: < 1 µs (on par with NVIDIA NDR, better than mainstream RoCE)

For messages > 4 KB, three-hop latency is ~63% lower than RoCE solutions

Multi-hop paths maintain "every hop fast, no slowdown across the full path"

"Compute-Storage-Transport" Tight Coupling: The Token Super-Highway

scaleFabric builds a three-layer tight-coupled acceleration channel atop native lossless RDMA:

Compute layer: IBGDA removes CPU coordination latency and bottlenecks.

Storage layer: Full storage-direct stack — NVMe over RDMA, XDS (XPU Direct Storage), and NFS over RDMA — lets GPUs/accelerators access remote NVMe pools or file storage directly, cutting storage access latency by >80%, sharply reducing TTFT and boosting token throughput during inference.

NVMe over RDMA : GPU directly accesses remote NVMe storage pool via RDMA network.

XDS (XPU Direct Storage) : Accelerator bypasses CPU to access storage.

NFS over RDMA : Runs traditional NFS protocol on RDMA network.

Core Value: MoE Communication and Token Generation Efficiency

MoE models are becoming mainstream. Their All-to-All communication (Dispatch + Combine) involves massive fine-grained exchanges. In the decode phase, each generated token may traverse dozens of MoE layers, each requiring two cross-node communications. Though messages are tiny, they are extremely latency-sensitive; any microsecond delay accumulates and amplifies.

IBGDA shines precisely here: GPU-initiated NIC communication delivers generational advantage in small-packet, high-frequency, strong-synchronization scenarios — single small-message bandwidth jumps significantly, All-to-All latency drops sharply, and the traditional "small-packet penalty" (small-message latency far exceeding large-message latency) is eliminated.

NVIDIA's Hybrid-EP communication library validates this direction, using IBGDA with RDMA+NVLink hybrid networking to approach hardware bandwidth limits. scaleFabric has completed adaptation with the DeepEP framework, meaning it can provide high-performance underlying network support for mainstream MoE models like DeepSeek in distributed communication.

Further, in disaggregated prefill/decode (PD-separated) inference architectures, IBGDA accelerates KV Cache movement between prefill and decode nodes. AMD's MORI framework has already verified this path: RDMA operations launched directly from GPU kernels move KV Cache across nodes completely bypassing the CPU. scaleFabric's capability promises substantial TTFT and TPOT improvements in long-context inference.

Conclusion: Communication Is the New Core Competitiveness

Competition at 10K+ GPU scale is no longer a pure compute arms race. As GPU compute keeps climbing, communication capability determines whether that compute can be unleashed. By letting GPUs fully own communication, IBGDA removes the legacy CPU bottleneck and opens new optimization space for end-to-end token generation efficiency. scaleFabric, as China's first large-scale IBGDA product, proves that domestic intelligent computing clusters can build a complete token acceleration system featuring "GPU-autonomous communication, storage direct access, and lossless networking." On the road to 100K-GPU clusters, communication is not just a supporting component — it is the true core competitiveness.