Introducing ROLL: A Scalable, User‑Friendly RL Framework for Large‑Scale LLM Training

In recent years, reinforcement learning from human feedback (RLHF) has become a key technique for the post‑training stage of large language models (LLMs), improving alignment and expanding applications such as reasoning enhancement and agent interaction.

To meet the growing demand for an efficient, scalable and user‑friendly RL system, Alibaba’s Taobao Group and iOrange Technology have open‑sourced ROLL (Reinforcement Learning Optimization for Large‑scale Learning), a framework that supports models from small to 600B+ parameters.

Key Features

Multi‑task RL : built‑in tasks covering mathematics, code, general reasoning, open‑ended QA and instruction following, with dynamic sampling and data weighting.

Agentic RL : native support for multiple environments and agents, parallel execution and management.

Algorithm‑friendly : configurable baselines, reward normalisation, data‑mask strategies, and out‑of‑the‑box PPO, GRPO, Reinforce++ support.

Rich training/inference engines : integrates vLLM, SGLang, Megatron‑Core, DeepSpeed without code changes.

Elastic resource scheduling : Ray‑based distributed architecture with 5‑D MegatronCore parallelism (DP/TP/PP/CP/EP) for heterogeneous GPU clusters.

Fine‑grained rollout scheduler : sample‑level lifecycle management, asynchronous reward computation and early stopping.

Observability : built‑in wandb, swanlab and TensorBoard logging.

Design for Three User Groups

Technical pioneers : elastic scaling and fault‑tolerance on thousands of GPUs for 600B+ models.

Product developers : flexible configuration of reward functions, environments and sampling ratios.

Algorithm researchers : efficient single‑ or few‑GPU experimentation and easy customization of RL algorithms, rewards and environments.

Architecture Overview

ROLL receives a user‑defined RL data flow and configuration, then creates a distributed executor and a rollout scheduler that coordinates workers and resources. The AutoDeviceMapping module allocates GPU/CPU resources from a resource pool to each parallel worker.

During the generation stage, the rollout scheduler feeds prompts to the actor model, which may interact with environment workers for multi‑turn tasks, while reward workers compute signals for dynamic sampling. In the inference stage, critic, reward and reference models perform forward passes, and in the training stage the actor and critic update their parameters with the computed rewards.

Experimental Results

On Qwen2.5‑7B‑base and Qwen3‑30B‑A3B‑base, ROLL improves overall accuracy from 0.18 to 0.52 and from 0.27 to 0.62 respectively, a 2.9× and 2.3× gain, without model collapse. In agentic environments such as Sokoban, FrozenLake and WebShop, success rates increase dramatically (e.g., Sokoban validation from 13.3% to 35.2%).

ROLL has already attracted over 1,000 stars on GitHub and continues to evolve with upcoming support for Qwen2.5‑VL Agentic RL, one‑step asynchronous pipelines, FSDP2, DeepSeekV3 and more.