NeurIPS 2026: Ascend RL Training System Hits 222% Speedup with Adaptive Scheduling

Background: Generation Becomes the Bottleneck in LLM RL Training

Reinforcement learning (RL) is now a core step for improving large language model (LLM) alignment and reasoning. Each policy update requires a full cycle: rollout generation, reward scoring, log-prob computation, and back-propagation. As output lengths grow, rollout generation dominates total training time.

The mainstream hybrid loop runs inference then training on the same devices. This structure is simple and keeps samples close to the current policy, but it has clear drawbacks:

Training GPUs/NPUs sit idle during generation, and vice versa.

Repeated mode switching forces re-sharding of the same hardware.

The two phases have fundamentally different system characteristics: back-propagation is compute-bound and favors large-scale synchronous parallelism, while generation is memory-bandwidth-bound and hits a scaling ceiling quickly when adding resources to a single inference instance.

Decoupling Only Solves Half the Problem

Asynchronous, decoupled pipelines let generation and training run concurrently, addressing the idle-resource issue. However, simply separating them does not automatically make multi-instance rollout generation efficient and stable.

When multiple actors decode responses of varying lengths, the system must simultaneously achieve two coupled goals: high utilization inside each inference minibatch, and a stable producer–consumer relationship between rollout generation and training (mediated by a Training Queue). The real challenge is how to schedule rollouts so that asynchronous generation is both fast and controllable .

Three Types of Bubbles Identified

The authors analyze bubbles in current rollout and training processes and classify them into three categories:

Padding bubbles : Within a minibatch, prompts produce responses of different lengths. Shorter responses finish early but the actor remains occupied by the longest response, so actual useful token generation is far below the padded compute volume.

Skewness bubbles : LLM output lengths follow a long-tail distribution. A few extremely long generations collapse the effective batch size in the later decoding stages.

Sorting bubbles : Sorting prompts by predicted length mitigates the first two problems but introduces a third. If short minibatches are dispatched first and long ones later, the generation rate varies over time — the Training Queue fills rapidly then drains slowly, causing queue-level instability.