Co-Fabric: Cross-Host Bus Interconnect Unifies xPU Memory for Trillion-Parameter AI
Inspur's Co-Fabric architecture introduces a bus-like scale-up interconnect that breaks host-domain boundaries, providing global unified addressing, native load/store semantics, and hardware P2P forwarding across servers, achieving 10-20% of RoCE latency for small messages and 30-80% higher inference throughput in 64-xPU tests.
Background: AI Super-Node Interconnect Challenges
As large model parameter counts reach trillions, a single xPU can no longer hold the full model weights, KV-cache, and optimizer states. Models must be split across multiple xPUs spanning multiple host domains. At this scale, inter-xPU communication efficiency directly determines overall AI system compute utilization. While chip compute continues to surge, data sharing has become the bottleneck — xPUs often sit idle waiting for remote data.
Two Existing Camps and Their Limitations
Vendors are exploring two main interconnect approaches: network-based (RoCE/InfiniBand) and bus-based (NVLink). Both exhibit fundamental shortcomings for modern AI workloads like MoE expert parallelism, tensor parallelism, and agent inference that demand massive fine-grained, high-frequency cross-card interaction.
Message semantics, not memory semantics: RoCE uses message passing. Accessing remote VRAM requires memory registration, rkey generation, and QP management — huge software overhead. Load/store instructions cannot directly address remote memory, causing severe semantic impedance and latency degradation for frequent small messages.
Fragmented address spaces, isolated host domains: Each host maintains an independent address space with no cross-host unified addressing. Cross-host access requires address translation and message encapsulation, forcing developers to handle physical topology awareness and manual data migration.
Protocol stack overhead cripples small-packet latency: RoCE packet send/receive incurs fixed encapsulation and scheduling overhead. MoE All-to-All and inference KV-cache synchronization generate massive small packets; protocol processing latency is amplified, causing xPUs to spin while waiting for data — a system-level "memory wall" that prevents compute utilization.
Co-Fabric: A Third Path – Cross-Domain Bus Scale-Up
Inspur's Co-Fabric (融脉) architecture, detailed in the paper Co-Fabric: Breaking Host-Domain Boundaries for Unified xPU Interconnection , proposes a new bus-like Scale-Up interconnect that breaks host-domain boundaries. It adopts an "interconnect-centric" design, no longer relying on a host CPU as the topology root. Co-Fabric unifies CPUs, memory, GPUs, VRAM, and other heterogeneous devices into a single interconnect domain, delivering:
Global unified addressing across hosts
Native load/store memory semantics
Hardware P2P forwarding bypassing CPUs
Link-layer native reliability
This eliminates the two core pain points: traditional bus interconnects "cannot cross hosts" and Ethernet RDMA "suffers from fragmented addressing and message-semantic overhead."
Core Technical Features
1. Simplified Four-Layer Protocol Stack
Co-Fabric discards the TCP/IP four-layer and OSI seven-layer models designed for wide-area internet. It compresses the stack bottom-up into four layers: Media Layer (ML), Link Layer (LL), Fabric Layer (FL), and Semantic Layer (SL). By removing independent transport and network layers, encapsulation overhead is eliminated. Routing and reliable transmission are pushed down to the link and fabric layers, implemented in hardware. Packet pipelining processes frames unidirectionally without software buffering, achieving sub-microsecond to microsecond processing latency — ideal for high-frequency small-packet traffic inside super-nodes.
Media Layer (ML): Handles physical bit encoding/decoding, shielding copper/optical media differences, compatible with PCIe physical links and mainstream CPU/xPU hardware.
Link Layer (LL) – Reliability Core: Implements credit-based flow control + link-level retransmission. Error recovery is confined to a single hop; faults are resolved locally without upper-layer end-to-end retransmission, avoiding large-scale latency jitter. This forms the lossless transmission foundation.
Fabric Layer (FL) – Cross-Domain Routing Core: Packet headers embed Domain ID (OS domain) + Port ID (globally unique port number). All routing information is encapsulated in the header; switches parse header Port IDs directly for forwarding without control-plane involvement. Supports single-packet hardware multicast, broadcast, and In-Network Compute (INC) with native INC-ADD, INC-MIN, INC-MAX. AllReduce collective communication aggregates in the forwarding path, reducing round-trip packets. On power-up, firmware performs topology auto-discovery, assigns globally unique Port IDs, and builds forwarding tables. Header format remains fixed at runtime; tables scale linearly with ports — no packet format changes as scale grows. Packets bypass host CPUs and OS network stacks entirely, achieving true hardware P2P cross-host forwarding.
Semantic Layer (SL): Directly exposes load/store and atomic operation primitives — the key differentiator from Ethernet RDMA's message model. Remote memory access maps directly to memory read/write instructions, eliminating message copy and message interface overhead.
2. Cross-Domain Port-ID Header-Embedded Routing
Traditional bus addressing binds to a single host's PCIe configuration space and cannot cross hosts; RDMA relies on software-managed complex queue pairs. Co-Fabric uses a header-embedded Port-ID forwarding mechanism:
At system power-up, firmware performs topology auto-discovery, assigning globally unique Port IDs to every Fabric port (xPU ports, switch cascade ports) and a Domain ID to each host domain. Switches generate forwarding tables once; they remain stable during operation.
Source writes target Domain-ID and Dst Port-ID into the Fabric layer header. Packets enter the switch fabric; each switch parses only the FL header ID and forwards based on Port-ID — no host CPU involvement, no OS network stack traversal.
Response packets return along the reverse path. Supports unicast, multicast, broadcast, and in-network reduction.
Result: cross-host xPU-to-xPU direct data movement without CPU mediation. As scale grows, header format stays constant; only switch forwarding table entries grow linearly, enabling elastic scaling to thousands of xPUs.
3. Global Unified Address Space + Shadow-Device Auto-Enumeration
This is Co-Fabric's most critical software experience innovation, solving cross-host distributed development complexity.
Flat Global Address Space: All VRAM within the super-node (local memory, each xPU's HBM, FPGA memory, remote host memory) is carved into non-overlapping global address segments. Any host sees the entire cluster's xPUs mapped into its local physical address space. Accessing a remote xPU's VRAM is simply accessing a local memory address range — no distinction whether the xPU is physically local or in another rack.
Shadow-Device Auto-Enumeration: The large number of cross-host xPUs visible to a host are not real hardware but virtual shadow devices simulated by the switch. On power-up, the switch discovers downstream physical xPUs, assigns each a fixed global HBM address range, and injects virtual device enumeration info to every host. All hosts obtain an identical device view and address layout, making each host "see the entire super-node's xPUs as if they were plugged into its local bus."
Business logic accesses the shadow device's address range; the switch intercepts packets, translates them into Co-Fabric protocol, and routes via Port-ID to the remote physical xPU. This is transparent to xPU chips, drivers, and applications — no xPU hardware or driver modifications needed.
Developers can use single-machine multi-card programming paradigms for distributed multi-host super-node workloads. No topology-aware code, no memory registration, no rkey/QP/CQ management, no explicit data migration. This drastically lowers development and tuning barriers for trillion-parameter model distributed inference and MoE training. At runtime, there is no RDMA memory registration software overhead; address mapping happens in hardware, keeping access latency at hundreds of nanoseconds to microseconds.
4. Link-Layer Native Dual Reliability
As link counts explode, physical link error rates rise sharply. Co-Fabric pushes end-to-end retransmission down to the link layer via two coordinated mechanisms:
Credit Flow Control: Sender and receiver synchronize receive buffer credits in real time. Sender transmits only when sufficient credits are available, preventing receive buffer overflow and packet loss at the source.
Link-Level Selective Retransmission (ACK/NAK): Every packet carries a sequence number and CRC. When a single link detects an error, only that link locally initiates retransmission. Error correction does not propagate across the cluster.
Credit updates and ACK/NAK reuse link control frames, consuming no business bandwidth. Local errors stay local; fault recovery latency is constrained to microseconds, avoiding long-tail jitter from end-to-end retransmission — perfectly matching AI training/inference deterministic latency requirements.
5. Engineering Implementation: Southbound and Hybrid South-North Architectures
The paper describes two hardware form factors supporting Inspur's Yuan Brain super-node products:
Southbound-Only Architecture: xPUs have only southbound Co-Fabric ports; CPUs stay out of the Co-Fabric data plane. Intra-domain uses CLOS full-mesh; cross-domain can use symmetric CLOS or Rail-Only asymmetric topologies. Simple architecture, suited for xPU-compute-centric scenarios.
Hybrid South-North Architecture: Both CPUs and xPUs possess southbound and northbound Co-Fabric ports. CPUs join the unified Fabric interconnect domain as peers alongside all xPUs, providing higher path redundancy and bandwidth. The paper's 64-xPU 3D-Mesh test system uses this hybrid architecture: dual-cabinet deployment, intra-node full-mesh, cross-node Rail-Only topology, balancing performance and hardware cost.
Measured hardware advantages vs. same-scale RoCE clusters: Co-Fabric links avoid Ethernet NIC signal rate losses; interconnect hardware replaces expensive Ethernet NICs and switches with Retimers + Co-Fabric switches, reducing interconnect hardware cost by 80% and link power by 5%.
Experimental Results: 64 xPU 3D-Mesh vs RoCE Baseline
Basic Communication Performance (AllReduce): For small messages (<4 MB), latency is only 10-20% of RoCE, bandwidth reaches 5-15x RoCE. For medium messages (4-64 MB), latency drops >50%, bandwidth improves 2-5x.
Scaling Linearity: Scaling from 8 to 64 cards, LoRA fine-tuning performance degrades only 7%, QwQ-32B full-parameter mode degrades only 2% — demonstrating excellent strong scaling.
Large Model Inference (DeepSeek R1): Inference throughput improves 30-80% over RoCE; at 128 concurrent requests, throughput reaches 1.8x RoCE. Breaking down Prefill and Decode stages, AllGather and AllReduce collective communication time drops 75% — communication overhead reduction is the core performance driver.
Conclusion
In the trillion-parameter model era, super-node performance ceilings are no longer dictated solely by xPU compute but by interconnect data movement efficiency. Co-Fabric charts a new path: it retains bus interconnect's native load/store memory semantics while breaking host-domain hard boundaries, building a unified Scale-Up super-node interconnect domain across servers and cabinets. It achieves goals previous solutions could not simultaneously deliver:
True cross-host, cross-OS-domain global unified address space — distributed AI workloads reuse single-machine programming models, drastically cutting MoE training/inference development and tuning costs, freeing developers from RDMA memory registration and queue management burdens.
Hardware-implemented nanosecond cross-host P2P forwarding with link-layer native reliability and in-network compute offload for collectives. Compared to RoCE, it dramatically reduces small-packet latency, increases effective bandwidth, and lowers overall interconnect TCO.
Protocol and physical topology decoupling — supports both small-scale full-mesh for extreme low latency and elastic switch-based scaling to thousands of xPUs, accommodating diverse training and inference workloads.
Measured results confirm Co-Fabric's protocol innovations deliver clear leads in basic collective communication, end-to-end large model inference, and large-scale scaling linearity tests, providing a new interconnect implementation approach for domestic large-scale AI super-nodes.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
