Why RoCE v2 Is Outpacing InfiniBand for Modern Data Centers

What Is RoCE v2?

RoCE v2 (RDMA over Converged Ethernet version 2) is an RDMA protocol designed for low‑latency, high‑throughput data transfer over Ethernet. By enabling direct memory access between systems, it minimizes CPU involvement and reduces communication latency, making it ideal for high‑performance computing (HPC), data‑center, and cloud environments.

Building on RoCE v1, RoCE v2 adds improvements that overcome the limitations of its predecessor and fully leverages converged Ethernet infrastructure, allowing traditional Ethernet traffic and RDMA traffic to coexist on the same network.

RoCE Network Interface Card

The RoCE NIC (Network Interface Card) is a specialized adapter that efficiently supports RDMA operations. By offloading RDMA tasks from the CPU, it significantly reduces data‑transfer latency and boosts overall system performance.

High‑performance switches increasingly adopt advanced forwarding chips such as Broadcom’s Tomahawk 3 and the newer Tomahawk 4, underscoring the importance of fast, large‑capacity packet processing in modern data‑center networks.

RoCE v2 vs InfiniBand Comparison

Physical Layer Architecture

RoCE v2: Operates on existing Ethernet infrastructure, allowing storage and regular data traffic to share the same network, which simplifies integration with current data‑center designs.

InfiniBand: Uses a proprietary communication fabric that typically requires dedicated cabling and specialized switches, increasing deployment complexity.

Protocol Stack and Compatibility

RoCE v2: Implements RDMA over Ethernet and integrates seamlessly with the TCP/IP stack, ensuring compatibility with standard network protocols.

InfiniBand: Employs a custom, high‑performance protocol stack that often necessitates specific drivers and configuration adjustments.

Switching Mechanism

RoCE v2: Runs on Ethernet switches that support Data Center Bridging (DCB), providing lossless Ethernet transmission.

InfiniBand: Relies on dedicated InfiniBand switches engineered for the lowest latency and highest throughput.

Congestion Management and Control

RoCE v2: Leverages DCB features of Ethernet switches for congestion management, creating a lossless environment, but does not include native congestion‑control algorithms.

InfiniBand: Offers built‑in congestion management (e.g., credit‑based flow control) and adaptive routing algorithms that dynamically adjust paths to prevent congestion.

Routing and Topology

RoCE v2: Uses standard Ethernet routing protocols such as RIP or OSPF, and its topology is constrained by the underlying Ethernet fabric.

InfiniBand: Provides specialized routing optimized for low latency, supporting multiple topologies like Fat‑Tree, hyper‑cube, and multi‑path configurations, enabling highly scalable and resilient networks.

When choosing between RoCE v2 and InfiniBand, the decision hinges on existing infrastructure, performance requirements, and budget. RoCE v2 excels at integrating with current Ethernet deployments for cost‑effective upgrades, while InfiniBand remains the preferred choice for ultra‑low‑latency, highly scalable HPC scenarios that justify dedicated hardware.

UEC Announces a New Transport Protocol

On July 19 2023, the Ultra‑Fast Ethernet Consortium (UEC) was founded by industry leaders including AMD, Arista, Broadcom, Cisco, HPE, Intel, Meta, and Microsoft. Recognizing that traditional RDMA struggles with the growing and heterogeneous traffic of AI/ML workloads, UEC is developing a modern transport protocol that incorporates RDMA features while delivering better load balancing and resource efficiency.

Conclusion

RoCE v2 plays a pivotal role in the RDMA ecosystem, offering a flexible, cost‑effective solution for high‑performance, low‑latency data transfer. Combined with the UEC’s upcoming transport protocol, RoCE v2 adapts to a wide range of applications from HPC to cloud computing. Nevertheless, organizations must evaluate their specific needs and existing infrastructure when selecting the optimal RDMA technology.