Overview of OpenFabrics Enterprise Distribution (OFED) and InfiniBand Software Architecture

OpenFabrics Enterprise Distribution (OFED) is a collection of open‑source software drivers, kernel code, middleware, and user‑level interfaces that support InfiniBand fabrics.

The OpenFabrics Alliance (OFA) released the first OFED version in 2005; it develops, tests, and promotes the stack to provide high‑efficiency messaging, low latency, and maximum bandwidth with minimal CPU overhead.

Mellanox OFED bundles drivers, middleware, user interfaces, and standard protocols such as IPoIB, SDP, SRP, iSER, RDS, and DAPL, offering Verbs programming interfaces and support for MPI, Lustre/NFS over RDMA.

The InfiniBand software stack is organized into three logical layers: the HCA driver layer, the core InfiniBand module, and the upper protocol layer. The middle layer provides services such as Communication Manager (CM), Subnet Administrator client, SMA, PMA, MAD service, GSI, Queue Pairs (QP), SMI, Verbs, and resource tracking.

CM – connection management services.

SA client – communicates with the Subnet Administrator.

SMA – subnet management agent.

PMA – performance management agent.

MAD – management datagram service.

GSI – generic service interface.

QP – queue pair handling for high‑bandwidth protocols.

SMI – subnet management interface.

Verbs – API for low‑level hardware operations.

InfiniBand supports IP‑based applications via IP over IB (IPoIB), allowing existing TCP/IP programs to run unchanged while gaining higher bandwidth and lower latency; higher‑performance socket protocols (SDP) and storage protocols (iSER, SCSI, NFS) are also available.

The OFED stack is available as ISO images for major Linux distributions and includes source code, binary RPMs, firmware, utilities, installation scripts, and documentation, and it is supported on Windows, various hypervisors, and major server operating systems.