How AI Workloads Are Redefining Network Architecture: Key Requirements and Topologies

This article, sourced from the "2025 AI Network Technology Whitepaper," analyzes the new demands AI workloads place on network infrastructure, covering massive‑scale training, high‑performance inference, and edge scenarios, and sets the stage for further technical discussion.

1. AI‑driven New Scenarios

As AI advances toward trillion‑parameter models, workloads shift from centralized training to distributed inference and edge‑centric acceleration, requiring deep network adaptation.

(1) Training Scenarios

Training large models involves iterative learning on massive datasets, with thousands of GPUs exchanging data east‑west. Even minor network bottlenecks or packet loss can significantly extend training time and affect model accuracy, demanding extreme performance.

(2) Inference Scenarios

Inference focuses on low‑latency, high‑throughput north‑south traffic between users and AI services, whether deployed in data centers, clouds, or edge devices, requiring flexible, efficient connectivity.

(3) Edge Scenarios

Edge AI pushes intelligence to devices like smart cameras and industrial sensors, reducing latency and bandwidth usage while enhancing privacy. Challenges include resource‑constrained environments, diverse connectivity (Wi‑Fi, 5G/6G, LoRaWAN), and complex deployments.

2. AI Network Requirements

To sustain AI’s growth, networks must break technical barriers and tightly integrate with AI algorithms and compute resources. Core requirements include:

(1) High Bandwidth & Low Latency

Ultra‑high bandwidth : Enables massive data exchange among compute nodes during training.

Ultra‑low latency : Critical for real‑time inference and reduces communication overhead in training.

(2) High Reliability & Stability

Lossless transmission : Prevents training interruptions and ensures data integrity.

Dynamic load balancing : Provides sub‑millisecond fault detection and recovery, maintaining continuous operation.

(3) Network Topology & Communication Patterns

Scaling to tens of thousands of GPUs demands flexible, extensible topologies and optimization for collective communication primitives such as AllReduce, AllGather, and ReduceScatter.

(4) Data Security & QoS

Resource isolation : Encryption, access control, and network slicing protect sensitive training data.

Traffic prioritization : QoS mechanisms ensure critical synchronization traffic receives bandwidth guarantees.

3. AI Cluster Network Topologies

Two primary topologies are discussed:

1. Fat‑Tree

Favoured for its efficient routing, scalability, and manageability. Small‑to‑medium GPU clusters use a two‑layer spine‑leaf design, while larger clusters adopt a three‑layer Core‑Spine‑Leaf architecture, accepting higher hop counts and latency.

GPU servers can connect via single‑rail (all NICs to one leaf) or multi‑rail (each NIC to a different leaf). Multi‑rail improves performance but increases impact of leaf failures.

2. Dragonfly

Designed for high‑performance computing, Dragonfly reduces network diameter to lower latency. It groups nodes into fully connected clusters, linking groups with a few high‑speed links, offering high scalability and lower cabling costs compared to Fat‑Tree.

Google’s Aquila data‑center uses Dragonfly with global link optimization, minimal hop count (max 3 hops), and virtual channels to avoid head‑of‑line blocking. However, limited software maturity and operational complexity can hinder adoption.

Overall, AI networks must deliver high bandwidth, ultra‑low latency, reliability, and scalability through optimized topologies, protocols, hardware acceleration, and fault‑tolerant designs to maximize compute utilization and support ever‑growing model and cluster sizes.