Why RDMA Outperforms Traditional Networking: A Deep Dive into DMA

What is DMA

DMA (Direct Memory Access) allows peripheral devices to read from or write to memory without CPU involvement. Without DMA, an I/O device such as a network card must request the CPU to copy data between memory and the device, consuming CPU cycles that could be used for computation.

With a DMA controller, the device initiates the transfer, and the controller moves data between memory and the I/O device while the CPU only starts and finishes the operation.

What is RDMA

RDMA (Remote Direct Memory Access) enables a node to directly read from or write to the memory of a remote node, bypassing the TCP/IP stack and minimizing CPU involvement. The data transfer is handled mainly by hardware, making the remote side unaware of the operation.

In traditional networking, data moves from the source memory to the destination memory via multiple CPU‑controlled steps, including protocol stack processing and multiple memory copies. RDMA eliminates most of these steps, allowing the NIC to transfer data directly between user‑space memory regions of the two nodes.

Advantages of RDMA

RDMA is widely used in high‑performance computing (HPC) and large data centers because it offers:

Zero‑copy : eliminates data copies between user and kernel space.

Kernel bypass : data moves directly between user memory and hardware, avoiding system calls and context switches.

CPU offload : remote CPUs are largely uninvolved in the data transfer, freeing them for other tasks.

These features provide higher bandwidth and lower latency, which are critical for bandwidth‑sensitive workloads (massive data exchange) and latency‑sensitive workloads (synchronizing compute nodes).

Protocols

RDMA technology is implemented through several protocols:

InfiniBand (IB) : a full‑stack protocol requiring dedicated IB hardware and switches.

RoCE (RDMA over Converged Ethernet) : uses Ethernet; RoCE v1 sits on IB’s link layer, while RoCE v2 uses UDP+IP, allowing routing over standard Ethernet networks.

iWARP : runs over TCP, offering better reliability on lossy networks but with higher overhead compared to RoCE and IB.

All three protocols share the same RDMA API but need specialized NICs for production use.

Players

Standards and Ecosystem Organizations

The InfiniBand Trade Association (IBTA) defines and maintains the IB standard, while the OpenFabrics Alliance (OFA) develops the open‑source OFED software stack that supports IB, RoCE, and iWARP.

Development Community

The Linux kernel includes an active RDMA subsystem, with contributions from hardware vendors such as Mellanox and Huawei. The user‑space community provides libraries like libibverbs and projects such as rdma‑core and perftest for building and testing RDMA applications.

UCX is a communication framework built on top of RDMA, offering higher‑level APIs for HPC workloads.

Hardware Vendors

Key vendors include Mellanox (now part of NVIDIA) and Huawei, both offering high‑speed RDMA‑capable NICs such as Mellanox’s ConnectX‑6 series and Huawei’s Kunpeng 920 with RoCE support.

Users

Major cloud and enterprise providers—Microsoft, IBM, Alibaba, JD.com—deploy RDMA in data centers and HPC clusters to achieve low‑latency, high‑throughput communication.