RDMA vs TCP/IP: Protocol Comparison and Overview of RDMA Types for Distributed Storage Networks

RDMA and TCP/IP

High‑performance computing and big‑data analytics demand I/O‑intensive, low‑latency communication that traditional TCP/IP stacks cannot satisfy because they rely on kernel‑mediated message passing, incurring significant data movement and copy overhead.

Remote Direct Memory Access (RDMA) allows network interfaces to read/write remote memory directly without OS kernel involvement, enabling high throughput and ultra‑low latency, especially suitable for large parallel clusters.

Figure 1‑1 – Comparison of RDMA and traditional TCP/IP

Types of RDMA

There are three main RDMA networks: InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP.

InfiniBand is a purpose‑built RDMA network offering high bandwidth and low latency but at a high cost. RoCE and iWARP are Ethernet‑based RDMA solutions that bring RDMA’s speed and low CPU usage to the ubiquitous Ethernet infrastructure.

RoCE has two versions: RoCEv1 operates at the Ethernet link layer and requires switches that support flow‑control (PFC); RoCEv2 runs over UDP/IP, adding IP‑level scalability.

Figure 1‑2 – RDMA network types

Table 1‑1 – Comparison of RoCE and InfiniBand

InfiniBand: Designed for RDMA from the hardware level, provides higher bandwidth and lower latency, but requires expensive IB NICs and switches.

RoCE: Ethernet‑based RDMA, consumes fewer resources than iWARP and supports more features; works with standard Ethernet switches if the NIC supports RoCE.

iWARP: TCP‑based RDMA that leverages TCP’s reliability; in large deployments it can create many TCP connections, increasing memory usage and requiring higher system specifications.

Common Network Protocols in Distributed Storage

IB – typically used for front‑end storage networks in DPC scenarios.

RoCE – commonly deployed in back‑end storage networks.

TCP/IP – used for general business networks.

For further reading on RoCE technology, see the linked articles:

Three Implementations and Applications of RoCE

High‑Performance Computing: RoCE Analysis and Applications

Choosing Between RoCE v2 and InfiniBand for HPC

Lossless Data‑Center Networking (IP, RDMA, IB, RoCE, AI Fabric)

NVMe over RoCEv2: Requirements and Test Specifications

Disclaimer: Thanks to the original author for the hard work. All reproduced articles are properly credited; please contact us for any copyright concerns.