Why NVMe over Fabrics Is Revolutionizing Modern Storage Networks

Background

NVMe (Non‑Volatile Memory Express) is a logical device interface designed for flash‑based SSDs, offering much lower latency than the legacy SATA interface, which was originally created for spinning hard drives in 2000.

NVMe Architecture

Introduced in 2011, NVMe defines a compact protocol with only 13 basic commands that can run on up to 64 K I/O queues, each holding 64 K commands, delivering performance up to three orders of magnitude higher than SATA. In client devices NVMe drives appear as M.2 or U.2 modules, while in servers they are often installed as PCIe expansion cards or M.2 boot devices.

Remote Block Storage Protocols

Two main approaches exist for remote storage: NAS (Network‑Attached Storage) for file‑level access using NFS or SMB, and SAN (Storage Area Network) for block‑level access. SAN relies on the SCSI protocol, which has been extended into iSCSI (TCP‑based), FCoE (Fibre Channel over Ethernet), and Fibre Channel NVMe (FC‑NVMe).

NVMe over Fabrics (NVMeoF)

NVMeoF started as a separate specification and later became part of the NVMe standard, allowing NVMe commands to be encapsulated over various transports (TCP, RDMA, Fibre Channel). This enables high‑performance, low‑latency block storage across network fabrics, similar to how iSCSI extends SCSI.

NVMe Transport Types

NVMe can be accessed via three main transport families:

Memory transport – direct PCIe or CXL/AXI connections to local storage.

TCP transport – NVMe/TCP uses a standard TCP connection for both commands and data, easy to deploy but incurs CPU overhead.

FC‑NVMe – runs NVMe over Fibre Channel, offering lower CPU usage and higher efficiency.

RDMA transport – NVMe/RDMA bypasses the kernel TCP/IP stack, allowing direct memory‑to‑memory transfers with minimal latency.

RDMA Overview

RDMA (Remote Direct Memory Access) enables applications and NVMe devices to read/write remote NIC memory buffers without involving the kernel network stack, reducing CPU load and latency. It is commonly implemented over InfiniBand or RoCE (RDMA over Converged Ethernet). RoCE v1 lacks IP encapsulation and cannot be routed, while RoCE v2 uses UDP for routable traffic. iWARP uses TCP for reliability but adds overhead.

InfiniBand – native high‑performance L2 network, ideal for HPC.

RoCE v1 – Ethernet with IB‑style packets, non‑routable.

RoCE v2 – UDP‑based, routable across IP networks.

iWARP – TCP‑based, reliable but higher CPU cost.

Conclusion

NVMeoF extends the high‑performance NVMe command set beyond local PCIe, providing a scalable, low‑latency solution for modern SAN environments. While TCP‑based NVMe is easy to adopt, RDMA and FC‑NVMe deliver superior efficiency, making NVMeoF the leading technology for data‑center storage networks.