Understanding RDMA: Principles, Advantages, and Implementation Details

RDMA (Remote Direct Memory Access) enables direct memory‑to‑memory data transfer between endpoints over a network without involving the operating system or TCP/IP stack, achieving microsecond‑level latency, high throughput, and minimal CPU usage.

Originally part of InfiniBand, RDMA has been extended to Ethernet via RoCE and iWARP, allowing deployment on widely used Ethernet infrastructures.

The development of RDMA is driven by the RDMA Consortium (RDMAC) and the InfiniBand Trade Association (IBTA), while the Open Fabric Alliance (OFA) defines the Verbs API and the OpenFabrics Enterprise Distribution (OFED) stack that provides cross‑platform support.

Key technical advantages include cut‑through switching, credit‑based flow control, hardware offload, and small buffers, which together reduce network latency to sub‑microsecond levels.

RDMA supports two communication models: two‑sided SEND/RECEIVE, which requires participation from both peers, and one‑sided READ/WRITE, which allows a host to directly access remote memory without remote software involvement, making it ideal for bulk data transfers.

Typical RDMA workflow involves creating Queue Pairs (QP) with Send and Receive Queues, registering Work Requests (WR) that become Work Queue Elements (WQE), and using Completion Queues (CQ) to notify the application of completed operations.

OFED provides both low‑level RNIC services and upper‑layer protocols (ULP) so existing applications can leverage RDMA without code changes.

In summary, RDMA’s success stems from its ability to dramatically lower processing latency and CPU overhead while delivering high bandwidth, and with Ethernet‑based RoCE and iWARP, it is poised for broader adoption in future data‑center and storage solutions.

InfiniBand’s low‑latency, high‑bandwidth design is a core foundation of RDMA.

RoCE and iWARP bring RDMA capabilities to Ethernet networks.

OFED and Verbs APIs enable seamless integration with existing applications.

Two‑sided and one‑sided operations provide flexibility for control and bulk data transfers.