An Overview of InfiniBand Technology: Architecture, Packet Structure, and Comparison with Ethernet

InfiniBand is one of the fastest‑growing high‑speed interconnect technologies, offering high bandwidth, low latency, and easy scalability; the article explores its packet structure, data transmission, hierarchy, comparison with Ethernet, switching mechanisms, and future development prospects.

1. Introduction With rapid CPU performance growth, high‑speed interconnect networks (HSI) have become critical for high‑performance computing (HPC). Gigabit Ethernet and InfiniBand dominate HSI, with InfiniBand being the fastest‑growing solution, developed under the InfiniBand Trade Association (IBTA) as a high‑performance, low‑latency technology.

2. InfiniBand Trade Association (IBTA) Founded in 1999 by merging two industry groups, IBTA is led by a steering committee from companies such as HP, IBM, Intel, Mellanox, Oracle, QLogic, Dell, and Bull. It focuses on compliance and interoperability testing and drives the evolution of InfiniBand specifications.

3. InfiniBand Overview InfiniBand provides a point‑to‑point switched fabric for processor‑to‑I/O communication, supporting up to 64,000 addressable devices. The InfiniBand Architecture (IBA) defines a standard, layered, point‑to‑point I/O framework used in servers, storage, and embedded systems. It offers universal low latency, high bandwidth, and low management cost, with subnets composed of end‑nodes, switches, links, and subnet managers.

InfiniBand inherits bus‑level bandwidth and latency advantages, implementing Direct Memory Access as Remote Direct Memory Access (RDMA). RDMA enables zero‑copy, kernel‑bypass data transfers, reducing CPU overhead and latency.

InfiniBand’s hardware components include Channel Adapters (CA), switches, routers, cables, and connectors. CAs are divided into Host Channel Adapters (HCA) and Target Channel Adapters (TCA). Switches interconnect subnets, and routers partition large networks into smaller subnets.

4. InfiniBand Packets and Data Transmission An InfiniBand packet consists of a Local Route Header (LRH), Global Route Header (GRH), Base Transport Header (BTH), Extended Transport Header (ETH), Payload (PYLD), Invariant CRC (ICRC), and Variant CRC (VCRC). The packet uses a 128‑bit IPv6‑style address for source (HCA) and destination (TCA) identification.

5. InfiniBand Architecture Layers The architecture is divided into Physical, Link, Network, and Transport layers. The Physical layer provides logical interfaces to the Link layer; the Link layer handles address, buffering, flow control, and error detection; the Network layer routes packets between subnets; the Transport layer adds transport headers and manages work queues via verbs.

6. Switching Mechanism InfiniBand employs a switched‑fabric topology with fault tolerance and scalability. Switches forward packets based on Destination Local Identifier (DLID), select output Virtual Lanes (VL) using SL‑VL mapping, perform credit‑based flow control, support unicast/multicast/broadcast, enforce partition keys, conduct extensive error checking, and arbitrate VL priority.

7. Comparison with Ethernet Compared with Ethernet, InfiniBand delivers significantly higher bandwidth (up to 168 Gbps for FDR) and lower latency, making it ideal for HPC workloads. Market data shows InfiniBand’s share in top‑100 HPC systems is rising while Ethernet’s share declines.

8. Conclusion InfiniBand is emerging as the preferred high‑speed interconnect, poised to replace 10/40 GbE in many scenarios. Future growth is expected in FDR, EDR, and HDR speeds, with bandwidth targets of 1 Tbps before 2020 and expanding applications in GPUs, SSDs, and clustered databases.