Overview of InfiniBand Technology: Development, Advantages, Architecture, Protocol Layers, and Applications

InfiniBand, an open‑standard technology, simplifies and accelerates connections between servers and also supports connections to remote storage and network devices.

Development of IB Technology

Specification drafting began in 1999 and the standard was officially released in 2000, but its adoption lagged behind Rapid I/O, PCI‑X, PCI‑E and Fibre Channel; Ethernet progressed from 1 Gbps to 10 Gbps faster. Consequently, InfiniBand Architecture (IBA) only saw widespread use in cluster supercomputers after 2005, and many of the Top‑500 supercomputers now employ IBA.

Major vendors such as Cisco, IBM, HP, Sun, NEC, Intel and LSI have joined or returned to the ecosystem, making InfiniBand one of the mainstream high‑performance interconnect technologies. To meet the I/O throughput demands of HPC, enterprise data centres and cloud environments, next‑generation 56 Gbps FDR (Fourteen Data Rate) and EDR InfiniBand have emerged.

Advantages of IB Technology

InfiniBand is widely used for FC/IP SAN, NAS and server‑to‑server connections; the iSER storage protocol (iSCSI over RDMA) has been standardized by the IETF. EMC’s entire product line has switched to InfiniBand networking, and IBM/TMS FlashSystem, IBM XIV Gen3, and DDN SFA series also adopt InfiniBand.

Compared with Fibre Channel, InfiniBand delivers roughly 3.5× higher performance, its switches have about one‑tenth the latency, and it supports both SAN and NAS.

Traditional FC SAN cannot satisfy modern storage‑system requirements. HP SFS and IBM GPFS are parallel file systems built on an InfiniBand fabric combined with iSER storage, fully breaking performance bottlenecks.

InfiniBand uses PCI serial high‑speed links (SDR, DDR, QDR, FDR, EDR) via Host Channel Adapters (HCAs), achieving micro‑second or even nanosecond latency, and employs link‑layer flow control for advanced congestion management.

InfiniBand implements Virtual Lanes (VL) for QoS; a single physical link can support up to 15 standard virtual lanes plus one management lane (VL15).

RDMA technology bypasses the kernel, providing remote read/write access and fully offloading CPU work; hardware transmission protocols ensure reliable delivery and higher performance.

Compared with TCP/IP, IB uses a trust‑based, flow‑controlled mechanism that ensures connection integrity, results in virtually no packet loss, and signals buffer availability after transmission, eliminating retransmission delays and improving overall efficiency.

TCP/IP can forward lost packets, but its continual acknowledgments and retransmissions significantly degrade performance.

Basic Concepts of IB

IB is a channel‑based, bidirectional serial transmission technology that employs a switched‑fabric topology; repeaters extend the link when needed. Each IB subnet can contain up to 65,536 nodes. IBA Switches and Repeaters operate within a subnet, while routers or gateways are required to interconnect multiple subnets.

Each node must connect to the IB subnet via an Adapter; CPUs and memory use a Host Channel Adapter (HCA), while disks and I/O use a Target Channel Adapter (TCA), together forming a complete IBA network.

IB media are flexible: backplane copper traces for intra‑chassis connections, copper cables (up to ~17 m) or optical fibers (up to ~10 km) for inter‑chassis links. Hot‑plug and active‑cable features provide auto‑detection and self‑adjustment.

IB Protocol Overview

InfiniBand is a layered protocol similar to TCP/IP, with each layer providing distinct services. It adopts the IPv6 header format; packet headers include the Local Route Header (LRH), Global Route Header (GRH), and Base Transport Header (BTH).

1. Physical Layer

The physical layer defines electrical and mechanical characteristics, including fiber and copper cables, connectors, backplane links, and hot‑swap properties. It specifies three physical port types (backplane, copper, optical) and details symbol encoding, framing, idle symbols, error checking, and synchronization.

2. Link Layer

The link layer describes packet format and operations such as flow control and intra‑subnet routing. It handles both management and data packets.

3. Network Layer

The network layer forwards packets between subnets, similar to the IP layer in traditional networks. It uses the Global Route Header (GRH) with IPv6‑style addressing (GID) to route packets across subnets.

4. Transport Layer

The transport layer manages message distribution, channel multiplexing, basic transport services, segmentation, and reassembly. It directs packets to specific queues (QP) and includes the Base Transport Header (BTH) with destination queue, operation type, sequence number, and partition info.

Receiving queues reassemble data into the appropriate buffers; all packets carry a BTH.

5. Upper‑Layer Protocols

InfiniBand supports several upper‑layer protocols, including SDP, SRP, iSER, RDS, IPoIB, and uDAPL.

SDP (Sockets Direct Protocol) enables existing TCP/IP applications to run over high‑speed InfiniBand.

SRP (SCSI RDMA Protocol) packages SCSI commands for RDMA‑based communication, allowing storage sharing across systems.

iSER (iSCSI over RDMA) transports iSCSI commands and data via RDMA on InfiniBand; it has been standardized by the IETF.

RDS (Reliable Datagram Sockets) provides a UDP‑like socket interface on InfiniBand, developed by Oracle.

IPoIB (IP over InfiniBand) makes InfiniBand networks compatible with TCP/IP applications transparently, offering higher bandwidth without code changes.

uDAPL (User Direct Access Programming Library) is a standard API that leverages RDMA for improved data‑center messaging performance, scalability, and reliability.

IB Application Scenarios

InfiniBand supports both direct‑attach and switched‑fabric topologies, making it ideal for high‑performance computing (HPC), large data‑center storage, and similar environments that demand sub‑10 µs latency, <10 % CPU utilization, and high bandwidth (commonly 56 Gbps or 100 Gbps).

On the host side, RDMA offloads CPU processing, reducing data‑path latency from tens of microseconds to about 1 µs. Meanwhile, InfiniBand’s high bandwidth (40 G, 56 G, 100 G), sub‑hundred‑nanosecond latency, and lossless characteristics combine Fibre Channel reliability with Ethernet flexibility.

9th China Cloud Computing Conference – For schedule details, ticketing and free professional tickets, see the linked article about the conference agenda.

Warm reminder: Please search “ICT_Architect” or scan the QR code below to follow the public account and receive more great content.