Why BBR Couldn't Dethrone CUBIC as Linux's Default TCP Congestion Control

BBR and CUBIC: Core Concepts

BBR's Model-Driven Approach

BBR (Bottleneck Bandwidth and Round-trip propagation time) is a model-driven congestion control algorithm. Unlike loss-based algorithms that treat packet loss as a congestion signal, BBR continuously estimates two key parameters: the bottleneck bandwidth (BtlBw) and the minimum round-trip time (RTprop). It then controls the sending rate to keep the in-flight data volume close to the bandwidth-delay product (BDP). Two control levers are used: pacing (packet sending rate) set to estimated bandwidth times a gain factor, and cwnd (in-flight data upper bound) set to BtlBw × RTprop × cwnd_gain. The goal is to avoid persistent queues, fill the bottleneck bandwidth, and achieve high throughput with low latency.

BBR State Machine

BBR cycles through four states:

State      | Goal                      | Method
STARTUP    | Quickly discover bandwidth | pacing_gain > 1, overshoot until bandwidth saturates
DRAIN      | Drain queue                | pacing_gain < 1, reduce in-flight to BDP
PROBE_BW   | Probe for higher bandwidth | Periodically adjust pacing_gain, continuously update BtlBw
PROBE_RTT  | Refresh minimum RTT        | Temporarily lower cwnd to avoid queue delay contaminating RTprop

CUBIC's Loss-Based Design

CUBIC is a loss-based congestion control algorithm that uses cwnd as the primary control variable. Without loss, it grows cwnd according to a cubic function; upon detecting loss, it multiplicatively reduces the window. Designed for high bandwidth-delay environments, it has been Linux's long-standing default congestion control algorithm.

Scenario Comparison

Where BBR Excels

Scenario                              | Reason
Cross-ocean/large file transfer (high BDP) | Rapidly and continuously approaches link bandwidth, not slowed by large RTT
Mobile/wireless links (4G/5G/Wi-Fi)  | Random loss does not trigger backoff, stable bandwidth utilization
Latency-sensitive services            | Pacing + cwnd ≈ BDP avoids persistently filling queues

Where CUBIC Excels

Scenario                              | Reason
General internet services (coexisting with many legacy TCP) | AIMD family, predictable coexistence with Reno/CUBIC, good compatibility
Paths with middleboxes/traffic shaping | Loss signal direct, not misled by rate estimation models
Maximizing fairness                   | Good fairness with similar algorithms, does not seize bandwidth

Key Trade-offs

Dimension          | BBR                              | CUBIC
Latency vs Throughput | Controls queue, reduces RTT variation | High throughput at expense of latency
Loss Interpretation   | Does not treat loss as congestion signal (tolerates random loss) | Loss equals congestion
Coexistence           | May be imbalanced in mixed environments | Conservative, good compatibility
Deployment Maturity   | Sensitive to pacing and queue management | Linux default, widely validated

Selection Recommendations

Cross-continental/high BDP transfers, mobile networks → try BBR first

General internet services, need fair coexistence with existing TCP → keep CUBIC

When uncertain, run A/B tests comparing throughput, 99th percentile RTT, loss rate

Deployment tip: BBR works better with FQ/FQ-CoDel

Why BBR Isn't Linux's Default

Four systemic shortcomings prevent BBR from becoming the default:

Fairness : Unfair when competing with CUBIC/Reno, can cause "bandwidth hogging" or "starving other flows" at network level.

Lossy Link Robustness : Does not use loss as congestion signal; on high-loss/jitter paths (VPN, tunnels, relays) causes excessive retransmissions and performance degradation.

Measurement Dependency : Relies on precise bandwidth/RTprop estimation; jitter, virtualization, CPU throttling cause estimation errors.

ISP Traffic Policing : Under token bucket policing, BBR's 1.25× gain probing persistently hits the ceiling, leading to extremely high retransmission rates.

BBRv2 and BBRv3 Optimizations

BBRv2 Improvements

Fairness: Better coexistence with loss-based algorithms, reduced unfair bandwidth contention.

Loss Response: Introduces limited loss tolerance, no longer completely ignores loss signals.

ECN Support: Enhanced ECN support for more proactive congestion control.

Probe Conservatism: ProbeBW phase more conservative, reduces queue buildup.

Convergence Speed: Faster adaptation to path changes.

BBRv3 Improvements

ECN Awareness: DCTCP-like ECN-driven control, avoids misjudgment in low-latency networks.

Probe Optimization: Probes adapt faster to network changes, ProbeRTT exits quicker (reduces short-flow startup latency).

Stability: Lower retransmission rate and slightly lower latency.

Fairness: More stable in multi-RTT scenarios (mixed with CUBIC/Reno).

Linux Kernel Congestion Control Framework (Linux 5.15.198)

Relevant directories: net/ipv4/,

include/net/

Sysctl/Procfs Configuration Parameters

1. tcp_congestion_control

Path: /proc/sys/net/ipv4/tcp_congestion_control
Permissions: 0644
Type: String
Description: System default congestion control algorithm
Handler: proc_tcp_congestion_control()

Usage:

# View default algorithm
cat /proc/sys/net/ipv4/tcp_congestion_control

# Set default algorithm
echo "bbr" > /proc/sys/net/ipv4/tcp_congestion_control

2. tcp_available_congestion_control

Path: /proc/sys/net/ipv4/tcp_available_congestion_control
Permissions: 0444 (read-only)
Description: List of all registered algorithms (space-separated)
Handler: proc_tcp_available_congestion_control()

Usage:

cat /proc/sys/net/ipv4/tcp_available_congestion_control
# Example output: cubic reno bbr dctcp vegas htcp ...

3. tcp_allowed_congestion_control

Path: /proc/sys/net/ipv4/tcp_allowed_congestion_control
Permissions: 0644
Description: List of algorithms allowed for non-privileged users
Handler: proc_allowed_congestion_control()

Usage:

# Allow only cubic and reno
echo "cubic reno" > /proc/sys/net/ipv4/tcp_allowed_congestion_control

Socket Option Configuration

TCP_CONGESTION

Option Value: 13
Macro Definition: include/uapi/linux/tcp.h:104
Description: Set/get socket's congestion control algorithm

Setsockopt implementation ( net/ipv4/tcp.c:3419-3437):

case TCP_CONGESTION: {
    char name[TCP_CA_NAME_MAX];
    if (optlen < 1)
        return -EINVAL;
    val = strncpy_from_sockptr(name, optval,
        min_t(long, TCP_CA_NAME_MAX-1, optlen));
    if (val < 0)
        return -EFAULT;
    name[val] = 0;
    lock_sock(sk);
    err = tcp_set_congestion_control(sk, name, true,
        ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN));
    release_sock(sk);
    return err;
}

Getsockopt implementation ( net/ipv4/tcp.c:4067-4075):

case TCP_CONGESTION:
    if (get_user(len, optlen))
        return -EFAULT;
    len = min_t(unsigned int, len, TCP_CA_NAME_MAX);
    if (put_user(len, optlen))
        return -EFAULT;
    if (copy_to_user(optval, icsk->icsk_ca_ops->name, len))
        return -EFAULT;
    return 0;

Algorithm Switching Mechanism

1. Algorithm Registration Flow

Module load (insmod tcp_cubic)
    │
    ▼
module_init() -> cubictcp_register()
    │
    ▼
tcp_register_congestion_control(&cubictcp)
    │
    ├── Check required functions (ssthresh, undo_cwnd, cong_avoid/cong_control)
    │   │
    │   └── Missing returns -EINVAL
    │
    ├── Compute hash key (jhash)
    │
    ├── Check key uniqueness
    │   │
    │   └── Duplicate returns -EEXIST
    │
    └── Add to tcp_cong_list global list
        │
        └── Success returns 0

2. Default Algorithm Selection Flow

System boot
    │
    ▼
late_initcall(tcp_congestion_default)
    │
    ▼
tcp_set_default_congestion_control(&init_net, CONFIG_DEFAULT_TCP_CONG)
    │
    ├── Find algorithm (tcp_ca_find_autoload)
    │
    ├── Try to get module reference
    │
    └── Set net->ipv4.tcp_congestion_control

Kernel config:

CONFIG_DEFAULT_TCP_CONG="cubic"  # Default congestion control algorithm

3. Socket Algorithm Assignment Flow

socket(AF_INET, SOCK_STREAM)
    │
    ▼
tcp_v4_init_sock()
    │
    ▼
tcp_assign_congestion_control(sk)
    │
    ├── rcu_read_lock()
    │
    ├── Get net->ipv4.tcp_congestion_control (default algorithm)
    │
    ├── bpf_try_module_get(ca, ca->owner)
    │   │
    │   ├── Success: set icsk->icsk_ca_ops = ca
    │   │
    │   └── Failure: fall back to tcp_reno
    │
    ├── Initialize icsk_ca_priv[]
    │
    └── Set ECN based on flags
        │
        ├── TCP_CONG_NEEDS_ECN -> INET_ECN_xmit()
        │
        └── Otherwise -> INET_ECN_dontxmit()

4. Runtime Algorithm Switching Flow (setsockopt)

User calls setsockopt(TCP_CONGESTION, "bbr")
    │
    ▼
do_tcp_setsockopt() -> tcp_set_congestion_control()
    │
    ├── Check icsk_ca_dst_locked
    │   │
    │   └── Locked returns -EPERM
    │
    ├── Find algorithm (tcp_ca_find_autoload)
    │   │
    │   ├── Found: ca = tcp_ca_find("bbr")
    │   │
    │   └── Not found: request_module("tcp_bbr") -> retry
    │
    ├── Check permissions
    │   │
    │   ├── ca->flags & TCP_CONG_NON_RESTRICTED -> allow
    │   │
    │   └── Otherwise: require CAP_NET_ADMIN
    │
    ├── bpf_try_module_get(ca, ca->owner)
    │   │
    │   └── Failure returns -EBUSY
    │
    └── tcp_reinit_congestion_control(sk, ca)
        │
        ├── tcp_cleanup_congestion_control(sk)
        │   │
        │   ├── Call old algorithm's release()
        │   │
        │   └── bpf_module_put(old algorithm)
        │
        ├── icsk->icsk_ca_ops = ca (new algorithm)
        │
        ├── memset(icsk_ca_priv, 0)
        │
        └── If connected -> tcp_init_congestion_control(sk)
            │
            ├── Call new algorithm's init()
            │
            └── icsk_ca_initialized = 1

5. Algorithm Invocation Flow

Packet ACK arrives
    │
    ▼
tcp_ack() -> tcp_cong_avoid()
    │
    ├── Check if congestion window limited
    │
    ├── Is slow start?
    │   │
    │   ├── Yes: tcp_slow_start()
    │   │
    │   └── No: call icsk_ca_ops->cong_avoid()
    │       │
    │       ├── CUBIC: cubictcp_cong_avoid()
    │       │
    │       ├── Reno: tcp_reno_cong_avoid()
    │       │
    │       └── DCTCP: tcp_reno_cong_avoid() (uses Reno)
    │
    └── Or call cong_control() (BBR)

Reference: https://gitee.com/shichaoyuan/learn-linux-with-ai