Why InfiniBand Is the Fastest Growing High‑Speed Interconnect for HPC

1. Introduction

InfiniBand is one of the fastest‑growing high‑speed interconnect technologies, offering high bandwidth, low latency, and easy scalability. With the rapid growth of CPU performance, high‑speed interconnects (HSI) have become critical for high‑performance computing (HPC). The main HSI technologies are Gigabit Ethernet and InfiniBand, the latter developing under the InfiniBand Trade Association (IBTA) as a high‑performance, low‑latency solution.

2. InfiniBand Trade Association (IBTA)

Founded in 1999, IBTA unites industry leaders such as HP, IBM, Intel, Mellanox, Oracle, QLogic, Dell, and Bull. It focuses on compliance and interoperability testing and drives the development and updates of the InfiniBand specifications.

3. InfiniBand Overview

InfiniBand provides a communication link between processors and I/O devices, supporting up to 64,000 addressable devices. The InfiniBand Architecture (IBA) defines a point‑to‑point switched I/O framework used in servers, storage, and embedded systems. It offers universal, low‑latency, high‑bandwidth, and low‑cost connectivity, with subnets composed of end‑nodes, switches, links, and subnet managers.

InfiniBand inherits bus‑level bandwidth and latency characteristics, implementing Remote Direct Memory Access (RDMA) to transfer data directly between memories without OS involvement, reducing bandwidth and CPU overhead.

Its advantages over Ethernet and Fibre Channel include standardized protocols, high data rates (up to 168 Gbps), RDMA support, transport offload, network partitioning, and multi‑level QoS.

4. InfiniBand Packets and Data Transfer

An InfiniBand packet consists of a Local Route Header (LRH), Global Route Header (GRH), Base Transport Header (BTH), Extended Transport Header (ETH), Payload (PYLD), Invariant CRC (ICRC), and Variant CRC (VCRC). The packet uses a 128‑bit IPv6‑style address for source (HCA) and destination (TCA) identification.

LRH : 8 bytes, identifies local source/destination ports, service level, and virtual lane.

GRH : 40 bytes, routes packets between subnets using IPv6 header format.

BTH : 12 bytes, specifies queue pair, opcode, sequence number, and segmentation.

ETH : 4‑28 bytes, provides reliable datagram service; Payload: 0‑4096 bytes of application data.

ICRC : 4 bytes, invariant error‑checking data.

VCRC : 2 bytes, variable error‑checking data.

5. InfiniBand Architecture Layers

The architecture consists of Physical, Link, Network, and Transport layers.

Physical Layer

Provides logical interfaces for the link layer, handling connectors, physical connections (copper or optical), power management, and encoding.

Establishes physical connections.

Notifies the link layer of connection status.

Delivers control and data signals to the link layer.

Link Layer

Manages packet transmission, addressing, buffering, flow control, error detection, and QoS.

Network Layer

Routes packets between subnets, supporting unicast and multicast operations.

Transport Layer

Adds transport headers, defines reliable and unreliable services, and manages work queues via QPs.

6. InfiniBand Switching Mechanism

InfiniBand uses a switched‑fabric architecture for fault tolerance and scalability. Switches forward packets based on the Local Route Header, perform VL arbitration, support partitioning, and provide management, performance, and board‑level agents (SMA, PMA, BMA). Key functions include output port selection, VL selection, credit‑based flow control, unicast/multicast handling, partition enforcement, error checking, and VL arbitration.

7. Comparison with Ethernet

InfiniBand surpasses Ethernet in bandwidth and latency, making it ideal for HPC. It also offers lower cost per performance. Market data shows InfiniBand’s share in top‑100 HPC systems is rising while Ethernet’s share declines.

8. Conclusion

InfiniBand is poised to replace 10/40 Gb Ethernet as the preferred high‑speed interconnect, with future growth expected in GPU, SSD, and clustered database applications. IBTA forecasts rapid market demand for FDR, EDR, and HDR technologies, targeting 1 Tbps bandwidth before 2020.