Why RDMA Is the Secret Engine Powering AI/ML Data Center Growth

Early RDMA and HPC

RDMA was first used mainly in high‑performance computing (HPC) clusters, where the focus was on supercomputing projects rather than cloud or enterprise data centers. At the end of 2022, the surge in AI/ML investment shifted data‑center spending, accelerating RDMA deployment.

By the end of 2023, the pace of RDMA network deployments exceeded the combined totals of 2021 and 2022, making RDMA a critical enabler for AI/ML expansion. The 650 Group forecasts the RDMA network market to surpass $22 billion by 2028 .

What Is RDMA?

Remote Direct Memory Access (RDMA) allows two servers to read and write each other's memory directly, bypassing CPUs, caches, and operating systems. This reduces latency, frees CPU cycles, and speeds up data transfer for networking, storage, and compute workloads.

RDMA in NICs

RDMA is implemented in the network interface card (NIC) of each server. By skipping the OS and network kernel, inter‑server performance improves. Although originally designed for large‑scale parallel HPC clusters, RDMA is now essential for AI/ML deployments that rely on massive parallelism.

RoCEv2 and Ethernet

RoCEv2 brings RDMA over Ethernet, using standard Ethernet infrastructure to carry RDMA traffic. Early RoCE versions required specialized Ethernet, but later versions run on ordinary Ethernet. The industry is heavily investing in congestion‑control mechanisms to reduce packet loss.

Ethernet Switch Scale

Data centers already have over 400 million Ethernet switch ports, so Ethernet will play an increasingly important role in AI/ML networks, with more RDMA operations moving onto Ethernet.

Server Market Shift

Customers are moving from general‑purpose servers to AI/ML‑optimized servers. By 2028, AI/ML server shipments are expected to rise from 1 million in 2023 to over 6 million, with a market size approaching $300 billion . Most of these servers will be equipped with high‑speed back‑end networks for inter‑node connectivity.

GPU Scaling and Data Transfer

Servers are increasingly populated with more GPUs (currently 8 per server is common, with 16‑32 GPU designs on the horizon) and larger GPU memory to train models ranging from billions to trillions of parameters. Efficient inter‑server data transfer, enabled by RDMA, becomes crucial for scaling these workloads.

Job Completion Time (JCT) Benefits

Direct memory access via RDMA speeds data delivery to GPUs, reducing Job Completion Time (JCT). Early AI/ML clusters suffered from idle GPUs due to packet loss or delayed data; RDMA mitigates these bottlenecks, improving overall performance compared with traditional networks.

NIC Market Landscape

All InfiniBand NICs support RDMA, but many Ethernet NICs still lack RoCE support. To stay competitive in AI/ML, Ethernet NIC vendors must add RoCE capabilities. As NIC speeds move beyond 400 Gbps, most Ethernet NICs are expected to support RoCE, driving up average selling prices.

AI/ML Backend Networks

Most AI/ML servers use a dedicated back‑end network, separate from the rest of the data‑center fabric. This network can be based on InfiniBand or Ethernet and focuses on high‑speed GPU‑to‑GPU or GPU‑to‑memory connections, expanding port counts and market potential.

Market Size Forecast

Before 2021, RDMA market size ranged from $4 billion to $7 billion annually, driven mainly by HPC. In 2023, AI/ML deployment spikes pushed demand above $60 billion, and the market is projected to exceed $220 billion by 2028.

Conclusion

RDMA and RoCE are essential for scaling AI/ML workloads; without them, data‑center expansion could not keep pace with rapid AI/ML growth. As the server market pivots toward AI/ML, RDMA and RoCE present a massive market opportunity, with both Ethernet and InfiniBand co‑existing and many customers deploying both technologies.