Understanding Ethernet Flow Control and Congestion Management (Part 1)

The chapter introduces Ethernet flow‑control, which comes in two IEEE‑standardized types: Link‑Level Flow Control (LLFC, IEEE 802.3x) that controls all traffic on a directly connected link, and Priority‑Based Flow Control (PFC, IEEE 802.1Qbb) that pauses only selected traffic classes while other traffic continues unimpeded.

Flow control is implemented with special pause frames. Each pause frame carries a quanta field (0‑65535) that specifies the pause duration. The pause time in seconds is calculated as (Pause quanta × 512) / Port speed (bits per second). For a 10 GbE link the maximum non‑zero pause time is 3.355 ms; for 100 GbE it is 0.355 ms.

When a receiver’s buffer utilization exceeds the pause threshold (XOFF), it sends a non‑zero pause frame to the sender, causing the sender to stop transmitting for the indicated duration. When utilization falls below the resume threshold (XON), a zero‑quanta pause frame (resume frame) is sent to restart traffic. The thresholds must satisfy pause threshold > resume threshold , leaving sufficient headroom (buffer space above the pause threshold) to avoid packet loss and sufficient footroom (buffer space below the resume threshold) to prevent under‑utilisation.

Lossless Ethernet devices maintain an ingress (input) lossless queue for each priority class. The queue size is limited by the device’s total buffer memory and the number of PFC‑enabled ports. Cisco Nexus 9000 switches, for example, support up to three lossless queues per port. The article shows how the ingress queue’s headroom (buffer size minus pause threshold) and footroom (resume threshold) affect congestion handling.

Configuration example for a Cisco Nexus 9000 switch demonstrates setting the buffer size, pause threshold, and resume threshold:

policy-map type queuing INPUT_Q
  class type queuing RDMA
    pause buffer-size 120000 pause-threshold 46000 resume-threshold 32000

In this example the buffer size is 120 KB, the pause threshold is 46 KB, and the resume threshold is 32 KB, leaving 74 KB of headroom. The article explains how to verify these settings with show queuing interface and discusses the impact of frame sizes (e.g., standard 1500‑byte Ethernet frames, FCoE up to ~2300 bytes, RoCE up to 2‑4 KB, and jumbo frames up to 9216 bytes) on headroom calculations.

For long‑distance PFC‑enabled links, additional buffer, pause‑threshold, and resume‑threshold adjustments may be required. The default settings on Nexus 9000 devices are sufficient for up to 100 m links, while devices such as the Nexus 93180YC‑FX can support 10 km FCoE links after tuning.

The article concludes that proper sizing of buffers, thresholds, and understanding of headroom/footroom are essential for preventing frame loss in lossless Ethernet storage networks and for achieving reliable congestion management.