How to Build a Multi‑Petabyte AI Super‑Cluster: Scaling Beyond Ten‑Thousand GPUs

1. Single‑Chip Capability

To support trillion‑parameter multimodal models, each GPU must improve compute performance and memory bandwidth. Strategies include adding more parallel cores, raising clock frequencies within power limits, optimizing cache hierarchies, adopting lower‑precision formats (e.g., FP8), and integrating domain‑specific accelerators. Memory should use high‑bandwidth, high‑capacity 2.5D/3D‑stacked HBM to reduce latency and enable massive model data placement.

2. Super‑Node Computing Power

Training massive models with long sequences and Mixture‑of‑Experts (MoE) architectures demands efficient All‑to‑All GPU communication. Recommendations:

Deploy servers that exceed the traditional 8‑GPU per node limit, enhancing scale‑up interconnects to boost tensor or MoE parallelism.

Integrate scale‑up‑capable switch chips inside nodes to increase point‑to‑point bandwidth and reduce latency.

Re‑architect GPU‑to‑GPU interconnect protocols (e.g., redesign packet formats, introduce CPO/NPO, increase SerDes rates, improve congestion control, and adopt multi‑chip C2C packaging) to raise bandwidth and lower latency for All‑to‑All traffic.

3. Heterogeneous Computing Fusion (CPU + GPU + DPU)

CPU‑centric data handling becomes a bottleneck at petascale. Offloading network‑related data processing to programmable DPUs provides hierarchical, low‑latency, unified control. A five‑engine DPU architecture is proposed:

Compute Engine: Exposes virtio‑net and virtio‑blk interfaces, abstracting vendor‑specific drivers.

Storage Engine: Implements TCP/IP or RDMA back‑ends for block, object, and file storage, handling all storage I/O on the DPU.

Network Engine: Moves virtual switch functions to the DPU, supports full‑line‑rate traffic, and integrates RDMA to achieve ~400 Gbps inter‑node bandwidth.

Security Engine: Uses root‑of‑trust mechanisms and standard IPsec encryption for tenant isolation.

Control Engine: Provides unified management of bare‑metal, VM, and container resources.

China Mobile’s “Panshi” DPU, originally FPGA‑based (2020) and upgraded to ASIC (2024), exemplifies this approach, enabling a CPU + GPU + DPU three‑platform cluster that eliminates compute islands caused by I/O bottlenecks.

4. Extreme Power‑Efficiency

Higher performance inevitably raises power density. Two optimization tracks are suggested:

Cooling: Adopt high‑density cold‑plate liquid‑cooling cabinets that house multiple liquid‑cooled GPU servers, dramatically improving space utilization over traditional air‑cooled racks.

GPU Chip: Move to advanced process nodes (7 nm or smaller), refine on‑chip bus designs, pipeline structures, voltage/frequency scaling, and clock‑gating. At the software level, employ fine‑grained monitoring and workload balancing to maximize energy‑efficient utilization.

5. High‑Performance Converged Storage

To sustain massive data movement, the cluster should adopt multi‑protocol, auto‑tiered storage that natively supports NFS, S3, and POSIX. Zero‑copy and zero‑conversion data paths enable “zero‑wait” model pipelines. A global file system can scale to >3 000 nodes, delivering >10 TB/s aggregate throughput, billions of IOPS, and >99.9999 % reliability, reducing checkpoint restore times from minutes to seconds.

6. Large‑Scale Reliable Networking

The network stack is divided into parameter, data, business, and management planes. The parameter plane—critical for model synchronization—requires massive scale, zero packet loss, high throughput, and ultra‑high reliability. Two mature technologies dominate:

InfiniBand (IB) and RoCE for low‑latency, lossless communication.

Emerging Ethernet‑based solutions such as Global‑Scheduled Ethernet (GSE) and Ultra‑Ethernet Consortium (UEC) that aim to close the performance gap.

Topology recommendations include Spine‑Leaf or Fat‑Tree designs, with full‑mesh inter‑connects between spine and leaf switches to ensure 1:1 up/down convergence.

7. Fault‑Tolerant Training (Checkpointing)

Large‑scale training relies on periodic checkpointing. To minimize downtime, multi‑level checkpoint storage (e.g., high‑speed memory tier plus asynchronous flush to persistent storage) reduces pause times and enables rapid recovery after hardware or software failures.