NVMe/TCP: Frequently Asked Questions and Answers

NVMExpress, Inc. recently announced the addition of NVMe over TCP (NVMe/TCP) to its NVMe transport series, marking a significant development in the NVMe ecosystem. The author previously hosted a webcast exploring the new specification; missed viewers can watch the recording via the provided link.

1. Will official documentation for NVMe/TCP be released when the NVMe 1.4 specification is finalized? Yes, the approved Technical Proposal TP 8000 is expected to be integrated into the NVMe‑oF 1.1 specification, with a release anticipated later this year.

2. What hardware, firmware, software, or drivers are required on the host side to support NVMe/TCP? No special hardware or firmware is needed. The host must run NVMe/TCP host software and the NVM subsystem, which are compatible with Linux Kernel 5.0, SPDK 19.01, and commercial NVMe/TCP target devices.

3. Is there any limit on the number of namespaces a host can have, and what resources are required? NVMe/TCP imposes no limit on namespaces; they are purely logical constructs that do not allocate host resources.

4. Does NVMe/TCP add latency for directly attached NVMe SSDs? Generally, NVMe/TCP does not introduce additional latency, though specific controller implementations may employ optimizations to avoid latency.

5. Which operating system kernels support NVMe/TCP? Linux kernel versions 5.0 and newer provide support for NVMe/TCP.

6. Will there be significant performance differences when running NVMe/TCP on a data‑plane network stack such as DPDK? On platforms with sufficient capabilities, running NVMe/TCP over the generic Linux network stack shows no fundamental performance gap, but DPDK‑based solutions can achieve better efficiency when CPU resources are constrained.

7. Is Data Center TCP recommended for NVMe/TCP workloads? Data Center TCP may offer superior congestion‑control algorithms, and NVMe/TCP already benefits from TCP’s flow‑control mechanisms; however, handling congestion in modern TCP/IP stacks remains a consideration.

8. Does NVMe/TCP support multiple R2T PDUs, and how do they compare to FC buffers? Controllers can send multiple lightweight R2T PDUs, subject to a host‑defined MAXR2T limit. The R2T credit mechanism is analogous to FC’s FCPBBC but operates at the NVMe command level.

9. How is traffic managed—only R2T and the standard TCP congestion window? End‑to‑end flow control is handled by TCP/IP, while NVMe transport‑level flow control uses the R2T credit mechanism.

10. In the submission queue, are multiple outstanding requests subject to PDU ordering constraints? No ordering constraints exist; PDUs for different NVMe commands are not required to follow a specific order.

11. How are patches and upgrades managed in NVMe/TCP, and are they non‑destructive? The protocol itself does not dictate patch or upgrade procedures; rollback strategies depend on vendor‑specific solutions.

12. Are there open‑source implementations of NVMe/TCP? Yes, both Linux and SPDK include open‑source NVMe/TCP target implementations.

13. Is there an equivalent implementation in iSCSI? iSCSI does not have a direct NVMe/TCP equivalent, though the concepts are similar—iSCSI transports SCSI over TCP/IP, while NVMe/TCP transports NVMe over TCP/IP.

14. How does NVMe/TCP’s performance (bandwidth, IOPS, latency) compare with NVMe/FC? No official benchmarks are available, but NVMe/TCP is expected to exhibit a modest performance reduction relative to direct‑connected NVMe and NVMe/FC.

15. Are there CPU utilization data comparing NVMe/RoCE and NVMe/TCP? No official data exist; NVMe/TCP typically consumes more CPU than NVMe/RDMA because the latter offloads parts of the transport to hardware.

16. What are the pros and cons of NVMe/TCP versus NVMe/RDMA, and are there performance differences? NVMe/TCP offers hardware‑agnostic scalability without requiring network‑infrastructure changes, while NVMe/RDMA can provide lower latency and CPU usage when hardware offload is available. Decision‑making should weigh performance, cost, and scale.

17. Can NVMe/TCP and NVMe/RDMA be combined on the same 100 Gb/s Ethernet network? Yes; NVMe/TCP runs over TCP/IP and NVMe/RDMA runs over RoCE or iWARP, both of which can share the same Ethernet fabric.

18. What is the maximum tolerable latency for NVMe PDU on Ethernet switches in distributed data‑center environments? NVMe/TCP does not define a maximum latency; the keep‑alive timeout defaults to two minutes, and network latency is generally not a limiting factor.

In summary, the NVMe/TCP transport binding specification is now publicly available, adding TCP as a new transport to the existing NVMe family (PCIe, RDMA, FC). It maps NVMe queues and NVMe‑oF encapsulation onto the IETF TCP protocol, optionally supporting features such as inline data integrity (DIGEST) and TLS. NVMe/TCP enables efficient end‑to‑end NVMe operations over standard IP networks, allowing large‑scale data centers to leverage existing Ethernet infrastructure while providing high performance and low latency. Open‑source implementations are available in Linux and SPDK, and drivers can be integrated directly into existing NVMe and NVMe‑oF software stacks.