Why RDMA Is Replacing TCP/IP in AI-Driven Data Centers

Background

With the rise of AI, deep‑learning server clusters and high‑performance SSD storage have pushed communication latency requirements to the microsecond level. Traditional TCP/IP stacks can no longer meet the performance needs of these systems.

Limitations of TCP/IP

The TCP/IP stack incurs tens of microseconds of latency because each packet transmission triggers multiple kernel context switches (5‑10 µs each) and at least three memory copies, all of which require CPU processing. This fixed overhead becomes a bottleneck for AI computation and SSD‑distributed storage, where sub‑microsecond latency is essential. Moreover, the CPU must repeatedly participate in packet handling, leading to sustained high CPU load, especially when network bandwidth exceeds 25 Gbps; in such cases, half of the CPU capacity may be consumed solely by data transfer.

Advantages of RDMA

RDMA’s kernel‑bypass mechanism allows direct data reads/writes between the application and the NIC, reducing end‑to‑end latency to around 1 µs. Its zero‑copy design eliminates intermediate memory copies, dramatically lowering CPU burden and improving efficiency. Benchmark data from a major internet provider shows that RDMA can increase computational efficiency by 6‑8×, and the 1 µs transmission latency enables SSD‑distributed storage to drop from millisecond‑level to microsecond‑level delays, making RDMA the default protocol in the latest NVMe over Fabrics implementations.

Current RDMA Deployment Options and Challenges

Two main RDMA transport solutions exist today:

InfiniBand : A closed‑architecture, vendor‑specific solution that cannot interoperate with existing IP Ethernet networks and suffers from vendor lock‑in and limited market share (less than 1% of Ethernet traffic).

RDMA over Ethernet (RoCE) : While compatible with standard IP networks, it lacks robust loss‑recovery mechanisms. Even a 2% packet loss can reduce RDMA throughput to zero; to maintain performance, loss must be below 0.001% (ideally zero). Congestion‑control mechanisms such as PFC and ECN can cause queue buildup, leading to PFC‑induced deadlocks and systemic network risk.

Impact of Distributed Architectures

Distributed computing introduces two traffic patterns that exacerbate network congestion:

Incast : Multiple nodes send data to a single receiver during the Reduce phase, causing sudden traffic spikes that can overwhelm the receiver’s interface.

Large‑packet interactions : As AI models grow, inter‑node messages can reach gigabyte sizes, increasing the likelihood of congestion and packet loss.

These patterns make dynamic latency (queueing and retransmission delays) dominate overall flow completion time, accounting for over 99% of total network delay.

Future Network Requirements

To satisfy AI‑era workloads, next‑generation data‑center networks must achieve three core goals: zero packet loss, ultra‑low latency, and high throughput. Huawei’s AI Fabric claims to meet all three by employing a proprietary congestion‑control algorithm that avoids the trade‑offs inherent in generic solutions like DCQCN, which require extensive per‑node parameter tuning.

Conclusion

RDMA’s ability to bypass the TCP/IP stack, combined with its near‑microsecond latency and zero‑copy data path, makes it the logical successor for high‑performance AI and storage workloads. However, achieving the full potential of RDMA requires an open, lossless Ethernet fabric and advanced congestion‑control mechanisms that can guarantee zero loss while maintaining low latency and high throughput.