Scaling Agentic Reinforcement Learning with a Decoupled T‑Architecture Using Verl and Argo Workflows

Agentic Reinforcement Learning (RL) Scaling Challenges

Agentic RL extends large language models from pure text generation to agents that learn through multi‑step trial‑and‑error in complex environments. The training loop consists of three stages: Rollout (experience collection), Reward (performance evaluation), and Training (parameter update).

Massive parallelism and scheduling: A single large experiment may require tens of thousands of independent rollout and evaluation tasks. Simple Python multiprocessing or basic task runners cannot express complex dependencies, conditional logic, retries, or dynamic resource allocation.

Reproducible isolated environments: Minor differences in library versions, system packages, or leftover files break reproducibility and can cause agents to converge to incorrect behaviours. Container‑level isolation of CPU, memory, and disk resources is therefore essential.

Decoupled “T‑Architecture” Using Verl and Argo Workflows

The architecture separates high‑level RL control from low‑level task orchestration:

Verl (macro control): Deploys policy models, defines policies and reward functions, and runs training algorithms such as PPO or GRPO.

Argo Workflows (micro execution): Executes massive parallel rollout, evaluation, and trajectory‑generation jobs as independent workflows.

Execution Flow

Verl deploys the initial policy to a dedicated inference service.

Verl triggers a batch of rollout jobs.

Each trigger creates tens of thousands of Argo Workflows; each workflow encapsulates the full lifecycle of a single agent.

During rollout, the workflow repeatedly queries the shared inference service for actions and executes them in the environment.

When rollout finishes, the workflow stores trajectories and evaluation metrics as Artifact s.

Verl asynchronously collects and aggregates the artifacts.

Verl computes rewards and updates the policy with PPO/GRPO, completing one training iteration.

Rollout Bottleneck and Benefits of the T‑Architecture

Empirical studies (MegaFlow, RhymeRL) show that rollout can consume 70‑90 % of total runtime in large‑scale Agentic RL. Traditional centralized solutions suffer from:

Limited concurrency – cannot sample at the required scale.

Insufficient isolation – shared environments cause dependency conflicts and state contamination.

Tail latency – a few slow tasks block progress and leave GPUs idle.

By delegating rollout to Argo Workflows, the T‑architecture achieves:

Container‑level isolation for reproducibility.

Distributed scheduling that eliminates resource contention.

Asynchronous execution that removes tail‑latency blocking.

“One Agent per Workflow” Paradigm

Each agent’s complete task is mapped to a single Argo Workflow consisting of three stages:

Rollout (interactive inference): The agent runs in a container (often co‑located with an execution container in one Pod), interacts with the environment over many steps, and records logs and trajectories as artifacts.

Evaluation: A dedicated container runs automated checks (e.g., test cases, static analysis) and outputs a score.

Post‑processing: Results are parsed, metrics extracted, and high‑quality experiences are written back to storage for the next training round.

Why Argo Workflows Instead of Ray for This Workload

Ray excels at low‑latency, online RL where actors need frequent communication. In the batch‑centric Agentic RL scenario described, tasks are independent and stateless, so the primary requirements are robustness, observability, reproducibility, and seamless cloud‑native integration. Argo Workflows provides a stateless, high‑throughput batch model, deep Kubernetes integration, and built‑in artifact handling, making it a better fit for the T‑architecture.

References

MegaFlow: Large‑Scale Distributed Orchestration System for the Agentic Era – https://arxiv.org/abs/2601.07526

Qwen3‑Coder‑Next Technical Report – https://huggingface.co/Qwen/Qwen3-Coder-Next

Argo Workflows – https://github.com/argoproj/argo-workflows

Verl (Volcano Engine RL) – https://github.com/volcengine/verl

RhymeRL: Efficient Reinforcement Learning for Large Language Models (2025).