How NVIDIA’s Blackwell Platform Redefines AI Supercomputing Networks

1. Blackwell Platform Network Configuration Analysis

From a networking perspective, NVIDIA introduces fifth‑generation NVLink, sixth‑generation PCIe, single‑port 800 Gb/s InfiniBand and Ethernet adapters, and 1.6 Tb/s optical module requirements. The upgraded capabilities are unlikely to be fully deployed on the latest DGX B200 servers immediately; they will be rolled out across subsequent products. The DGX B200 server is specified with eight NVIDIA B200 Tensor Core GPUs, four 800 Gb/s OSPF optical modules, and a maximum power consumption of 14.3 kW.

2. DGX GB200: NVLink Domain Scaling to 576 GPUs

The NVIDIA DGX SuperPOD built on the DGX GB200 system targets training and inference of trillion‑parameter generative AI models. Each liquid‑cooled rack houses 36 NVIDIA GB200 Grace Blackwell super‑chips (36 Grace CPUs and 72 Blackwell GPUs) interconnected via NVLink. Multiple racks connect through NVIDIA Quantum InfiniBand, allowing scaling to tens of thousands of GB200 super‑chips.

Within a single rack, a NVLink domain can contain either 36 or 72 GPUs. The rack comprises 18 compute nodes and nine L1 NVLink switches. Leveraging fifth‑generation NVLink, each switch provides 144 × 100 Gb ports, yielding a total of 1,296 × 100 Gb NVLink ports across the nine switches. Consequently, each Blackwell GPU receives 18 ports, delivering a combined NVLink bandwidth of 1.8 Tb/s.

GB200 NVL72 employs a custom copper‑cable cartridge to connect the nine L1 NVSwitches with the 18 compute nodes. NVIDIA states that the GB200 NVL72 can interconnect 576 GPUs within a single NVLink domain. The current GB200 NVL72 configuration includes 72 CX7 ports (OSFP 400 Gb/s InfiniBand) and does not use the newer CX8 NIC.

3. New InfiniBand Switch – Dual‑Layer Fat‑Tree for Massive GPU Clusters

The NVIDIA Quantum‑X800 Q3400‑RA is a 4U InfiniBand switch whose chassis size is twice that of the previous‑generation Quantum‑9700. It is NVIDIA’s first switch to adopt a 200 Gb/s‑per‑lane SerDes solution, offering 144 × 800 Gb ports via 72 OSPF 1.6 Tb optical modules. NVIDIA claims that, with this high‑performance switch, a dual‑layer fat‑tree topology can connect up to 10,386 network interface cards.

At the GTC conference, NVIDIA also introduced the Spectrum‑X800 SN5600, an Ethernet switch with a total capacity of 51.2 Tb ports, configurable with 64 OSPF 800 Gb optical modules.

4. Continued Need for 800 Gb/s and 400 Gb/s Optical Modules

The GTC presentation indicates that the Blackwell platform will bring fifth‑generation NVLink, sixth‑generation PCIe, and eighth‑generation InfiniBand networking. However, early‑delivery products still rely on 800 Gb/s and 400 Gb/s optical modules. The DGX B200 server uses four 800 Gb/s OSPF optical modules to provide eight 400 Gb/s ports, identical to the DGX H100 configuration. The B200 GPU‑to‑800 Gb module ratio mirrors that of the H100. The GB200 NVL72 requires 72 OSPF 400 Gb/s modules to enable IB networking for a ten‑thousand‑card cluster.

5. 1.6 Tb/s Optical Modules Required by New IB Network

The NVIDIA Quantum‑X800 Q3400‑RA and CX8 NIC explicitly require 1.6 Tb optical modules. Compared with the previous IB generation, the major change is the increase of port capacity to 115.2 Tb, allowing a dual‑layer fat‑tree topology to connect 10,386 NICs. This dramatically reduces the cost of IB networking for massive Blackwell GPU clusters, potentially weakening the price advantage of Ethernet solutions and strengthening the competitiveness of IB.

6. Expanded NVLink Domain Drives New Copper Cabling Requirements

Fifth‑generation NVLink bandwidth rises from 900 Gb to 1.8 Tb, and a single NVLink domain can now host up to 576 GPUs. The most significant change is the shift of the L1 NVSwitch from a PCB‑mounted chip on the server to nine cabinet‑level switches, increasing connectivity to 72 GPUs and necessitating copper cables for intra‑cabinet NVLink links.

While NVIDIA has not disclosed a concrete wiring scheme for connecting 576 GPUs, the capabilities of the latest IB switches suggest that an L2 NVSwitch will be required. The final choice between copper cables and optical modules hinges on whether copper can meet the distance requirements across eight cabinets; low‑cost copper solutions are likely to be preferred.