An Introduction to RDMA: Concepts, Advantages, Protocols, and Programming Basics

What is RDMA? Direct Memory Access (DMA) allows a device to read or write host memory without CPU involvement. Remote Direct Memory Access (RDMA) extends this capability to remote machines, enabling memory reads/writes without interrupting the remote CPU.

Traditional Network vs. RDMA In conventional networks, both sending and receiving ends rely on the CPU for NIC control, interrupt handling, and packet processing. With RDMA, the CPUs on both ends are largely bypassed; the NIC performs DMA transfers directly between user‑space buffers and the network, assembling and disassembling packets in hardware.

Core Advantages of RDMA

Zero‑copy: Data moves directly between application buffers and the network without traversing the software stack.

Kernel bypass: Applications issue transfers from user space without context switches.

CPU offload: Remote memory is accessed without consuming remote CPU cycles, reducing cache pressure.

Typical Scenarios RDMA is valuable when at least one of the following is required: ultra‑low latency (e.g., HPC, financial services, Web 3.0), high bandwidth (e.g., storage, backup, cloud computing), or minimal CPU utilization (e.g., HPC, cloud workloads).

Three RDMA Protocols

Infiniband: A native RDMA‑capable network protocol requiring specialized NICs and switches.

RoCE (RDMA over Converged Ethernet): Carries RDMA over Ethernet by encapsulating InfiniBand headers within Ethernet frames; requires RoCE‑capable NICs.

iWARP: Implements RDMA over TCP; can run on standard Ethernet hardware but may lose some performance benefits.

How to Use RDMA You need an RDMA‑enabled network adapter (e.g., Mellanox Connect‑X). The link layer can be Ethernet or InfiniBand, both supporting RDMA‑based applications.

RDMA Communication Flow Before a SEND‑RECV operation, the receiver posts a receive work request (RQ). The sender posts a SEND work request (SQ). Upon receipt, the remote NIC generates an ACK, which is reported via a Completion Queue (CQ) to the application, completing the transaction.

RDMA Programming Basics

Operations: SEND/RECV, WRITE/READ, ATOMIC, and shared‑RQ techniques.

Transport modes: RC (reliable connection, like TCP), UC (unreliable connection), UD (unreliable datagram, like UDP).

Key concepts: Send Request (SR), Receive Request (RR), Completion Queue (CQ), Memory Registration, Protection Domain, Scatter‑Gather entries, and polling mechanisms.

The article draws its content from the “RDMA Technology Research” material and provides links to additional RDMA resources and related white‑papers.