Skip to content
First 10K-Card AI Cluster Network Report: Domestic IB Breakthrough at 100K Scale

First 10K-Card AI Cluster Network Report: Domestic IB Breakthrough at 100K Scale

The article analyzes China's first ten-thousand-card intelligent computing cluster network technology report, detailing how Sugon's scaleFabric domestic InfiniBand native RDMA system overcomes scaling bottlenecks through hardware-native lossless RDMA, hierarchical supernode/supercluster architecture, large subnet expansion to 114,623 cards, and high-reliability multi-tenant operations, validated on the 100K-card 曙光8000 super-intelligent fusion cluster.

Architects' Tech Alliance
Architects' Tech Alliance
Architects' Tech Alliance

Background: Scaling Laws Drive 100K-Card Cluster Reality

Large model parameters reach 2–3 trillion by 2026 and are projected to hit 10 trillion by 2027. China's daily token calls surged from 100 billion (early 2024) to 140 trillion (March 2026), a 1,400× increase. Single compute-center peak compute expands 3.3× annually, making 100K-card and even million-card clusters an industrial necessity.

Report Identifies Network as Primary Bottleneck

The report Ten-Thousand-Card Intelligent Computing Cluster Network Technology Development and Domestic Research Report , edited by experts Shan Zhiguang, Zhang Yunquan, and He Baohong, states that moving from 10K to 100K cards is limited not by compute chips but by interconnect network scalability, lossless transmission, fault management, and engineering deployment. Domestic full-stack breakthroughs have achieved native IB RDMA technology and validated it at scale on the 曙光8000 (登峰) 100K-card super-intelligent fusion cluster.

Four Core Technical Demands for Domestic IB Networks

1. Hardware-Native Lossless RDMA for MoE Communication

MoE expert parallelism relies heavily on All-to-All communication, directly determining training throughput and decode latency (TPOT). Traditional Ethernet RoCE uses PFC pause for coarse-grained flow control, which at massive scale leads to packet loss, PFC storms, and deadlocks, requiring manual congestion-control tuning. IB employs credit-based hardware link-level flow control (CBFC) plus link-level retransmission (LLR), delivering true lossless transport at the hardware link layer without complex upper-layer tuning. It also fully supports memory-semantic Load/Store/Atomic operations for low-latency intra-supernode scale-up and message-semantic communication for cross-cabinet scale-out collective operations.

2. Scale-Up + Scale-Out Hierarchical Architecture

The architecture adopts a two-tier design: Supernode Scale-Up (high-density accelerator aggregation within a cabinet) and Supercluster Scale-Out (cross-cabinet Clos networking). Scale-Up addresses intra-supernode latency; as MoE expert parallelism grows, All-to-All latency and bandwidth become equally critical. International top solutions have expanded supernodes from 8 to 72 cards, but size must balance business benefit, hardware cost, cooling, and reliability risks. Scale-Out uses supernodes as building blocks, interconnected via multi-plane Clos to support 10K to 100K+ cards.

3. Large Subnet Expansion Beyond Traditional Limits

Traditional IB caps a single subnet at 49,151 cards, forcing 100K-card clusters into multiple subnets. The industry adopts Multi-Plane and Multi-Rail architectures with three-layer Clos and high-radix switch chips to achieve a single plane of 100K cards and a future million-card cluster. Efficiency relies on adaptive routing, Packet Spray, topology-aware optimization, and collective communication offload. scaleFabric expands the IB address space by 2.33×, enabling a single subnet to support 114,623 cards — a 133% capacity increase — while remaining compatible with pre-training, SFT/RLHF post-training, and KV-Cache sharing for high-throughput inference.

4. High Reliability + Multi-Tenant SLA for Training-Inference Integration

At 100K cards, link failures become a probabilistic norm: single-link hourly failure probability is 0.00258%, yielding ~2.7 link errors per hour. The network requires autonomous SerDes with continuous BER tracking, differentiated design for multiple link types, millisecond failover, and recovery time that does not grow with cluster scale. Operations demand rapid onboarding of 100K NICs, proactive unhealthy-node detection and isolation, full-stack observability, and business-aware scheduling. Business-wise, inference off-peak hours accommodate training tasks to maximize hardware ROI; the network must carry mixed training, long-context, and multi-agent inference traffic with multi-tenant isolation and differentiated QoS.

scaleFabric: Full-Stack Domestic IB Native RDMA System

Sugon's scaleFabric covers self-developed 112G SerDes IP, switch chips, NICs, switches, and platform software — six core chips including domestic processors, GPU accelerators, inter-chip interconnect switches, 400/800G switch chips, 400G NIC chips, and PCIe 5.0 switch chips. Compared to international top-tier IB products, scaleFabric matches Write/Read/Send bandwidth and latency as well as GDR path performance, achieving benchmark parity with supply autonomy . It has undergone long-term stable operation in 10K-card commercial clusters. The design abandons pure port-bandwidth chasing; instead it balances convergence ratio, domain scale, and hop count under constraints (application demands, chip specs, XPU compute, power, cooling, signal reliability, wiring cost, maintainability) to maximize system throughput and token production efficiency.

Core Chips: Switch and NIC Dual Breakthrough

scaleFabric400 Switch Chip: 64 Tbps bidirectional switching capacity, VCT mechanism, 260 ns forwarding latency; 40× 800G or 80× 400G ports; self-developed 112G PAM4 SerDes; single subnet supports 114,623 cards (2.33× traditional IB limit).

scaleFabric400 NIC Chip: 400 Gb/s bandwidth, built-in self-developed RDMA engine, 0.93 μs end-to-end latency, measured throughput ≥390 Gb/s, ultra-large QP capacity for MoE expert parallelism; domestic 112G PAM4 SerDes fully benchmarks overseas high-end NICs.

Signal integrity is ensured by self-developed SerDes plus continuous BER tracking improvement, with differentiated link designs to support large-scale wiring and cabinet-level supernode high-density coupling.

Hierarchical Networking: Supernode + Supercluster

Supernode Scale-Up: Domestic inter-chip interconnect switch chips build a level-1 Clos low-latency fabric, supporting 8/40/64/192-card supernode tiers that aggregate accelerators into a low-latency compute domain meeting MoE EP parallel All-to-All demands.

Supercluster Scale-Out: Using supernodes as units, scaleFabric 400 switches construct a three-layer multi-plane Clos network. A single plane supports 10K+ cards; the cluster evolves to 100K+ cards with high-radix switches reducing cost. Immersion phase-change liquid cooling raises cabinet density; 50 m fiber reaches 10K accelerators.

Future roadmap: SerDes rates from 112G to 224G/448G; optical interconnect from pluggable modules/LPO to NPO/CPO co-packaged optics, moving optical engines closer to chips to break electrical distance/power constraints for million-card clusters.

Network Operations: Managing 100K Cards as One System

The Fabricview management platform provides unified control across scaleFabric, IB, and RoCE networks, exposing OpenAPI and OpenMetric interfaces. Capabilities include asset/topology auto-discovery, monitoring views with link heatmaps, real-time/historical alarms, config check and firmware repository, one-click/scheduled inspections, and a knowledge-base/rule-driven fault-repair closed loop.

Engineering Validation: 曙光8000 100K-Card Super-Intelligent Fusion Cluster

曙光8000 is China's first super-intelligent fusion system on the native fusion technical route, now a core node of the National Supercomputing Internet and the premier landing case for domestic IB network technology.

System Positioning

It natively fuses supercomputing and intelligent computing, supporting major scientific tasks ("four orientations, four extremes") and general workloads — scientific/engineering computing, AI for Science, AI/large models, cloud services — on a single system. Since launch, it processes >15K jobs daily, has deeply optimized 300+ key applications across 30+ scientific and industrial fields, validating domestic high-speed network real-world capacity at 100K-card scale.

scaleFabric Value in 曙光8000

The entire 100K-card cluster runs on scaleFabric domestic IB native RDMA, completing the domestic chain from intra-supernode inter-chip interconnect to cross-cabinet supercluster networking. Communication bottlenecks concentrate on two collective operations: training All-Reduce (tests bandwidth and convergence ratio) and MoE All-to-All (tests latency and lossless capability). Daytime serves high-throughput token inference; nighttime inference troughs absorb large-model post-training and pre-training, with mixed training-inference scheduling maximizing compute asset ROI.

Ecosystem Compatibility

scaleFabric supports mainstream communication library interfaces, adapts multi-brand AI accelerators, and is compatible with mainstream AI computing software stacks. Network optimization is driven by actual application behavior — e.g., MPI interface call statistics across 80 LAMMPS scenarios guide collective communication and topology-aware tuning. As a National Supercomputing Internet core node, it hosts massive models, agents, and datasets covering large models, bioinformatics, materials chemistry, meteorology/ocean, oil exploration, commercial aerospace, and digital twins.

Industry Insights and Future Evolution

Domestic high-speed interconnect has completed 10K-card validation and entered 100K-card engineering deployment, but faces challenges: higher-rate SerDes iteration, CPO co-packaged optics engineering, million-card multi-plane networking, large-scale fault self-healing, and network optimization under mixed training-inference loads. Four evolution directions:

Hardware Iteration: SerDes to 224G/448G, ports 400G→800G→1.6T, switching capacity 51.2T→102.4T, optics LPO→NPO→CPO to resolve bandwidth/power/distance trade-offs.

Architecture Upgrade: Optimize multi-plane Clos with adaptive routing, Packet Spray, topology-aware optimization, and in-network collective offload (Collective Offload) to cut MoE/RLHF communication overhead.

Software & Operations: Enhance topology-aware scheduling, fault self-healing, multi-tenant differentiated SLA; unified management already compatible with scaleFabric/IB/RoCE via OpenAPI/OpenMetric, adapting to sustained large-scale token inference growth.

Cross-Domain Collaboration: Leverage National Supercomputing Internet to explore Scale-across cross-data-center networking, extending 100K-card local clusters to cross-regional compute services; one network supports both supercomputing and intelligent computing, avoiding duplicate-build TCO. Open supernode/supercluster to multi-brand AI accelerators; jointly build AI Computing Open Architecture Joint Laboratory with industry partners.

Conclusion

As model parameters advance toward 10T and Agent-driven token traffic grows exponentially, 100K-card intelligent computing clusters have moved from concept to reality. High-speed interconnect is the "neural center" of compute clusters; domestic IB native RDMA is the key path for ultra-large-scale clusters. Credit-based flow control determinism, single-subnet 100K-card capacity, and millisecond fault recovery independent of scale are capabilities that upper-layer tuning cannot补齐. scaleFabric's engineering practice on the 100K-card super-intelligent fusion cluster fills the domestic IB native lossless RDMA gap, returning high-end RDMA supply, evolution, and ecosystem to an autonomous controllable rhythm. Continued refinement of chips, protocols, optical interconnect, and application ecosystems will enable domestic high-speed interconnect to support million-card intelligent computing infrastructure, solidifying the compute foundation for new quality productive forces.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

network architectureMoERDMAInfiniBandsupercomputingscaleFabricAI clusterlossless networking
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.