Why Network Becomes the New Bottleneck for AI Training and How InfiniBand vs RoCE Compare

AI large‑model training is built on distributed GPU clusters, which create huge amounts of inter‑node communication. This communication load makes the network the critical performance bottleneck for AI compute, especially as current data‑center networking technologies are largely dominated by foreign vendors and suffer from generational gaps.

InfiniBand vs RoCE

Both InfiniBand and RoCE are RDMA‑enabled protocols introduced by the IBTA to provide high‑performance data transfer. InfiniBand, first defined in 1999, offers hardware‑level guarantees of low latency and high throughput with dedicated switches, NICs, optical modules, and fiber. It is a closed ecosystem with high procurement and maintenance costs and cannot interoperate with Ethernet.

RoCE, introduced in 2010, adapts RDMA to Ethernet by encapsulating it at the NIC level. It relies on lossless Ethernet features such as PFC to achieve lossless transmission, but its scalability and forwarding performance are limited compared to InfiniBand, and it remains an open, industry‑wide solution.

Current Challenges

Existing Ethernet‑based forwarding and scheduling mechanisms have inherent deficiencies for AI model training workloads. Simple optimizations of upper‑layer protocols cannot overcome these limitations; instead, fundamental changes to the underlying forwarding and scheduling logic of network chips are required to break the lossless Ethernet performance ceiling.

In summary, the rapid growth of AI training workloads exposes the inadequacy of conventional Ethernet networks, highlighting the need for either widespread adoption of InfiniBand or the development of next‑generation network chip architectures that can support the demanding bandwidth and latency requirements of modern AI systems.