Understanding RDMA: How QP, WQE, and Doorbell Power High‑Performance Communication

In high‑performance computing, distributed storage, and AI training clusters, RDMA (Remote Direct Memory Access) serves as a communication accelerator by bypassing the CPU, offering low latency and high throughput. Beginners often encounter terms such as QP (Queue Pair), WQE (Work Queue Entry), and Doorbell, which are the essential components that enable RDMA's efficiency.

1. RDMA Communication Basics: From CPU‑Mediated Transfer to Direct Hardware Access

Traditional TCP/IP networking requires the CPU to copy application data into kernel buffers, process protocol‑stack encapsulation, and then send it via the NIC, incurring multiple memory copies and context switches with latencies of tens of microseconds to milliseconds.

RDMA aims to let the NIC read and write remote memory directly, eliminating CPU involvement. To maintain safety and control, RDMA uses a predefined task‑queue system: the local host submits explicit operation commands (e.g., "read 1 KB from remote address X into local address Y") to the NIC, which then executes them in order.

2. QP (Queue Pair): The Scheduling Hub of RDMA

A QP is the smallest logical unit in RDMA, consisting of a Send Queue (SQ) and a Receive Queue (RQ). Each QP has a unique identifier (QPN) that the remote side uses to locate the target QP.

Send Queue (SQ) : Holds active operation commands (WQEs) such as send or remote read.

Receive Queue (RQ) : Holds descriptors for buffers where incoming data should be placed.

The QP progresses through a state machine similar to TCP's three‑way handshake: RESET (initial), INIT (basic transport attributes set), RTR (receive side ready), and RTS (ready to send). Both peers must reach the RTS state before data can flow, typically triggered by commands like ibv_modify_qp.

3. WQE (Work Queue Entry): The Detailed Instruction Set

A WQE is the minimal execution unit stored in an SQ or RQ. It contains structured fields that tell the NIC exactly what to do:

Opcode : operation type such as SEND, RDMA_READ, or RDMA_WRITE.

Target address : remote memory address obtained via RDMA's memory‑registration mechanism.

Local buffer : for SEND, the start address and length of local data; for RECEIVE, the destination address for incoming data.

Control flags : options like remote acknowledgment requirement or data alignment.

When an application (e.g., an MPI program) calls the Verbs API function ibv_post_send, it actually appends a WQE to the SQ. The NIC continuously polls the SQ, extracts the earliest WQE, and performs the specified operation. WQEs placed in the RQ serve the passive receive side.

4. Doorbell: The Fast‑Path Notification Mechanism

The NIC can process WQEs in nanoseconds, while the CPU can only submit them in microseconds. Waiting for the NIC to discover new WQEs would waste CPU cycles, but relying solely on NIC polling could increase latency if submissions are delayed.

The Doorbell mechanism resolves this mismatch. After a program writes a new WQE to the SQ or RQ, it immediately writes to a specific memory‑mapped I/O (MMIO) Doorbell register, conveying the queue identifier and the offset of the new WQE. The NIC detects this write, pauses any current task if necessary, and promptly processes the newly submitted WQE.

Typical workflow: the program calls ibv_post_send or ibv_post_recv, the WQE is placed at the tail of the queue, then the program writes the Doorbell register via MMIO, notifying the NIC of the pending work.