Understanding NVMe, NVMe‑oF, and RDMA for High‑Performance Storage Networks

Historically, network engineers did not worry about storage bandwidth because network speeds outpaced rotating hard‑disk drives, but the advent of ultra‑fast SSDs and NVMe has changed that landscape, making storage a potential network bottleneck.

NVMe, a protocol designed for PCIe‑connected non‑volatile memory, treats SSDs as memory rather than disks, offering multi‑queue support, deep queue depths, and NUMA awareness, which dramatically reduces I/O overhead compared to legacy SCSI.

The article outlines the evolution from SCSI, which views storage as devices with a single 64‑command queue, to NVMe, which provides up to 65,535 queues each with 65,535 commands, enabling each CPU core to communicate directly with storage.

NVMe‑oF extends NVMe beyond the host by allowing storage to be accessed over a network fabric. Three official transport bindings are covered:

NVMe/FC – uses Fibre Channel (or FCoE) without RDMA.

NVMe/TCP – runs NVMe over standard TCP/IP Ethernet.

NVMe/RDMA – leverages Remote Direct Memory Access over InfiniBand or Ethernet (RoCE, iWARP).

Each transport has distinct requirements: dedicated InfiniBand for NVMe/IB, Fibre Channel for NVMe/FC, and shared or converged Ethernet for NVMe/TCP, NVMe/RoCEv2, and NVMe/iWARP. The article discusses lossless vs. lossy Ethernet considerations, noting that RoCE (UDP‑based) prefers lossless fabrics, while iWARP and TCP can tolerate packet loss.

Practical guidance includes selecting appropriate NICs (e.g., Broadcom NetXtreme E, Mellanox Connect‑X) and understanding that RDMA offloads reduce CPU involvement and enable zero‑copy transfers, but bandwidth and latency remain critical.

Finally, the piece advises network architects to weigh dedicated versus shared Ethernet, evaluate latency budgets for specific applications, and consider proximity of storage targets to hosts to meet strict NVMe latency requirements.

References and further reading links are provided for deeper dives into DPDK, SPDK, and detailed RDMA implementations.