Why RDMA and RoCE Are Becoming Critical Enablers for AI/ML Deployments

Background and Market Shift

Historically, RDMA (Remote Direct Memory Access) was primarily used in high‑performance computing (HPC) clusters. Until the end of 2022, most HPC investments focused on supercomputing projects, with limited adoption in cloud or enterprise data centers. The emergence of AI/ML as a major investment focus caused a dramatic shift in data‑center spending, accelerating RDMA deployment.

By the end of 2023, the pace of RDMA network deployments exceeded the combined totals of 2021 and 2022. The 650 Group predicts that the RDMA network market will surpass US$22 billion by 2028 .

Technical Overview of RDMA and RoCE

RDMA enables two servers to read and write each other's memory directly, bypassing CPUs, caches, and operating‑system kernels. This reduces latency, frees CPU cycles, and speeds up data transfer for networking, storage, and compute workloads.

RDMA is implemented in the NIC of each server. RoCEv2 extends RDMA over Ethernet, allowing RDMA traffic to use standard Ethernet infrastructure. Industry efforts are improving Ethernet congestion‑control mechanisms to reduce packet loss, making Ethernet a viable carrier for RDMA at scale.

Server Market Evolution

Customers are moving from general‑purpose servers to AI/ML‑optimized servers. By 2028, the number of AI/ML servers is expected to rise from 1 million in 2023 to over 6 million, with a market size approaching US$300 billion . Most AI/ML servers currently host eight GPUs; future designs will likely accommodate 16‑32 GPUs per node.

As model parameters expand from billions to trillions, GPU memory capacities must grow, and efficient inter‑server data transfer becomes essential for scaling training workloads. RDMA’s ability to quickly move data to GPUs directly shortens Job Completion Time (JCT) and improves overall cluster utilization.

Performance Impact

Early AI/ML clusters suffered from idle GPUs due to network bottlenecks and packet loss. RDMA eliminates many of these issues, delivering lower latency and higher throughput, which translates into reduced JCT and better performance metrics compared with traditional Ethernet or InfiniBand alone.

NIC and Switch Market Dynamics

All InfiniBand NICs support RDMA, but many Ethernet NICs still lack RoCE support. Ethernet NIC vendors must integrate RoCE to stay competitive in the AI/ML market. As NIC speeds move beyond 400 Gbps, RoCE support is expected to become standard, driving up average selling prices (ASPs) for high‑performance Ethernet adapters.

Differences in RoCE implementation—such as offload engines, processor types, and R&D expertise—create multiple performance tiers among vendors. Ongoing product releases will narrow these gaps, improving interoperability and expanding customer choice.

AI/ML Backend Networks

AI/ML servers typically use a dedicated backend network, which can be based on InfiniBand or Ethernet with RoCE. This network focuses on high‑speed GPU‑to‑GPU or GPU‑to‑memory connections, supplementing the broader data‑center fabric and increasing port density.

Multiple backend networks may coexist within a single AI/ML deployment, allowing different vendors or technologies to address specific workload requirements.

Market Size and Forecast

Before 2021, the RDMA market ranged from US$0.4 billion to US$0.7 billion annually, driven mainly by HPC. In 2023, AI/ML deployments pushed annual demand above US$6 billion, with expectations to exceed US$22 billion by 2028.

Conclusion

RDMA and RoCE are essential for scaling AI/ML workloads; without them, the rapid growth of AI/ML deployments would be constrained by network latency and bandwidth limits. The server market’s pivot toward AI/ML creates a massive opportunity for RDMA technologies, and both Ethernet and InfiniBand will continue to coexist, often within the same AI/ML clusters.