How Libra Allocates Resources for Agentic RL Post‑Training and Boosts Throughput Up to 3×

Problem Context

Agentic RL post‑training generates trajectories whose lengths can differ by tens of times because models invoke external tools (search, code execution, etc.). The longest 10% of trajectories consume >50% of rollout time, and as the policy evolves the distribution of trajectory lengths drifts. Rollout latency grows roughly 95× when sequence length increases from 1K to 32K tokens, while training time grows only 3.9×, making rollout the dominant bottleneck in many steps. The iteration time is therefore T_iter = max(T_rollout, T_train), and a static GPU split between training and rollout quickly becomes imbalanced.

Libra Design Overview

Libra treats training and rollout as a coupled optimization problem under a fixed GPU budget. It consists of four core components:

Global Resource Planner jointly searches for (i) the GPU split between training and rollout, (ii) the training parallelism configuration (tensor‑, pipeline‑, data‑parallelism and MoE expert parallelism), (iii) the number and type of heterogeneous rollout inference instances (TP‑1, TP‑2, TP‑4, TP‑8), and (iv) the expected makespan of the configuration. The planner enumerates feasible training topologies with a topology‑aware decision‑tree, prunes based on memory and communication constraints, and uses a cost evaluator that models both training and rollout execution times.

Heterogeneous Rollout Cluster sorts incoming rollout requests by historical length and allocates them to TP instances via dynamic programming, achieving the optimal mapping between request intervals and heterogeneous TP resources.

C‑MLFQ (Causality‑Driven Multi‑Level Feedback Queue) builds a prefix tree from historical tool‑return events. Each node stores the distribution of remaining length for the suffix of the trajectory. At runtime, a request starts in the smallest TP bucket; after each tool return the system queries the tree using tool type, payload size, and status. The request migrates to a larger bucket only when both the mean and the 90th‑percentile of the remaining‑length distribution fall into that bucket. After trajectory completion the tree is updated offline.

Elastic Hybrid Pool enables low‑cost movement of workers between rollout and training without rebuilding the core communication group. Hybrid workers join as full data‑parallel replicas, preserving the core TP/PP topology. When a rollout worker switches to training it receives a zero‑gradient placeholder so that the All‑Reduce computation remains mathematically equivalent until the worker’s state aligns with the training checkpoint.

Dynamic Reconfiguration

Libra periodically reads the latest trajectory statistics, resolves a new global configuration, and triggers resource movement only if the estimated throughput gain exceeds the reconfiguration cost. This mechanism tracks workload drift while avoiding frequent oscillations.

Experimental Setup

Experiments run on six nodes with a total of 48 NVIDIA A800‑SXM4‑80GB GPUs (NVLink/NVSwitch intra‑node, 200 Gb/s RoCE inter‑node). The system uses GRPO, a maximum model length of 40960 tokens, and samples 16 trajectories per prompt. Three Agentic RL workloads are evaluated:

Search‑R1 – multi‑turn search queries with external knowledge.

R2E‑Gym – software‑engineering agent that invokes Bash, Python, etc., on real code repositories.

DAPO‑Math‑17K – 17 000 competition‑level math problems.

Baselines include verl‑Colocated, verl‑Static‑Uniform, verl‑Greedy‑Heuristic, and AReaL‑Static‑Optimal (the best static configuration derived from the initial workload).

Performance Results

Libra achieves the highest throughput on all three tasks, up to 3.0× the best static baseline. On Search‑R1 the average throughput is 2700 token/s, ≈63 % higher than AReaL‑Static‑Optimal, with a routing accuracy of 91.1 % (vs. 65.2 % for embedding‑based length prediction and 44.8 % for traditional MLFQ) and only 8.2 % of tokens migrated. Wall‑clock time to reach target rewards is reduced by up to 2.5× (17.9 h for Search‑R1, 26.7 h for DAPO‑Math‑17K, 63.2 h for R2E‑Gym).

Ablation on R2E‑Gym shows the contribution of each module (baseline Static‑Uniform throughput 423 token/s):

Global planning: +20 % (≈510 token/s)

Heterogeneous TP: +8 % (≈41 token/s)

C‑MLFQ: +22 % (≈115 token/s)

Elastic execution: +13 % (≈97 token/s)

Combined, Libra reaches 763 token/s, an overall gain of ~80 % over the static uniform baseline.

Implementation Details

Libra comprises ~13 k lines of Python and C++/CUDA code. The training loop builds on the verl framework, inference uses vLLM, and model parallelism is provided by Megatron‑LM. The open‑source repository is hosted at https://github.com/NetX-lab/Libra and includes Slurm and non‑Slurm quick‑start scripts, data‑preparation utilities, configuration examples, observability guidance, and full experiment scripts.

Key Insight

In Agentic RL the rollout bottleneck is not static; as the policy and trajectory distribution evolve, the true bottleneck shifts between rollout and training. Libra’s combination of global resource planning, heterogeneous execution, causality‑driven scheduling, and elastic resource swapping enables the system to continuously approach the optimal balance and sustain high throughput throughout long‑running training.