From LLM Rollout to Agentic Rollout: Design Insights and Lessons for an Agentic RL Training Framework

1. From LLM Rollout to Agentic Rollout

Reinforcement‑learning (RL) training alternates between rollout (data generation) and model update. In the reasoning era of LLMs a rollout is a single turn: prompt → response → reward. Agentic RL requires multiple model calls and tool interactions (e.g., reading a repository, fixing code, testing), so a rollout must manage many rounds of inference and environment manipulation.

2. Coupled Architecture

Principle

Early implementations embed the entire agent loop inside the rollout module: the rollout calls the model, parses output, invokes tools, and feeds results back until termination.

messages = [task_prompt]
while True:
    response = llm_server.generate(messages)
    trajectory.record(messages, response)
    if response.is_final_answer():
        final_answer = response.content
        break
    tool_result = env.execute(response.tool_call)

Advantages

Agent behavior is fully transparent and easy to debug because the rollout controls model calls, tool execution, and termination.

Training trajectories are straightforward to collect since all interactions occur inside the rollout.

Drawbacks

Limited extensibility: adding a new agent often requires a custom harness.

Black‑box agents (e.g., Claude Code) cannot be fully simulated, causing a mismatch between training and real‑world usage.

3. Decoupled Architecture

To avoid simulating agent logic, the decoupled design runs the agent natively while the rollout system only orchestrates its lifecycle and records the communication between the agent and the LLM server.

Key Challenges

Running and managing the agent’s environment.

Collecting trajectories without direct access to the agent’s internal state.

Coordinating a complete rollout (resource creation, execution, result aggregation).

Core Components

Controller : receives tasks, creates sessions, coordinates Runtime Manager and Gateway, tracks status, aggregates results.

Runtime Manager : creates isolated sandboxes, prepares files/dependencies, starts/stops the agent, cleans up resources.

Gateway : sits between the agent and the LLM server, forwards requests, records request/response pairs, builds multi‑turn trajectories.

Agent : runs in the sandbox and sends inference requests to the Gateway.

LLM Server : performs model inference.

The flow: Controller creates a session → Runtime Manager provisions a sandbox → Agent runs and communicates via Gateway → Gateway records the full message‑level trajectory → Controller merges agent output, trajectory, and evaluation signals into a training sample.

4. Complete Agentic Rollout Process

Resource Creation & Initialization

Task reception : Controller receives a Task containing prompt, agent configuration, and evaluation rules.

Session creation : Controller asks Gateway to create a unique Session ID.

Sandbox preparation : Runtime Manager creates an isolated environment, writes task files, clones the code repository, installs dependencies, and injects Prompt, Session ID, and Gateway address.

Agent Execution & Trajectory Collection

Agent starts inside the sandbox.

Each model call is sent to Gateway, which records the request (messages, tool calls) under the Session ID and forwards it to the LLM Server.

Gateway stores responses, tool‑call metadata, and token usage, then returns the reply to the Agent. This loop repeats, building a complete Trajectory for the session.

Because recording happens at the communication layer, Gateway does not need to understand the agent’s internal state.

Result Aggregation & Resource Release

After the agent finishes, Runtime Manager collects logs, output files, and any error information.

Controller fetches the full trajectory from Gateway, filters out failed traces, and runs the evaluation defined in the Task to produce a reward.

Controller aligns Task input, trajectory, agent result, and reward into a training sample, then destroys the Session and releases sandbox resources.

5. Gateway Design Details

Protocol Normalization

Gateway supports OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. An adapter converts incoming requests to the unified OpenAI Chat Completions schema and wraps LLM server responses into the appropriate streaming or non‑streaming protocol.

Request Scenario Classification

Main Agent Loop : user prompt, system prompt, history, tool definitions → full task reply.

Context Compression : compresses long histories before the next turn.

Sub‑Agent Call : delegates subtasks to specialized agents.

Session Summary : generates a concise summary for later retrieval.

Heartbeat : lightweight ping to check provider health.

Gateway tags each request with its scenario, allowing downstream components to treat high‑value training data (main loop, sub‑agent) differently from low‑value signals (summary, heartbeat).

Trajectory Reconstruction

Each request is independent; Gateway reconstructs a full execution trace by prefix‑matching the messages array. When context compression breaks monotonic growth, Gateway stores multiple partial trajectories and later merges them based on Session ID and logical continuity. Strict string matching is relaxed for agents that mask variable names (e.g., replacing API keys with ***).

Token Consistency

Training requires both the textual message array and the exact token‑id sequence used during inference. Re‑tokenizing stored text can change token IDs, breaking KV‑cache reuse and misaligning log‑probability data. Gateway therefore records the original token IDs returned by the LLM server and performs tokenization/detokenization internally to keep inference and training data perfectly aligned.

6. Runtime Manager Design Details

Abstract Layering

Runtime Manager separates two orthogonal concerns:

Agent Harness : describes how to configure, launch, and parse a specific agent (e.g., Codex, Claude Code, Hermes).

Sandbox Backend : describes how to create and control the execution environment (E2B sandbox, Docker container, local process).

New agents only need a new harness implementation; new environments only need a new backend, without touching the other layer.

Runtime Manager
├── Agent Harness
│   ├── CodexHarness
│   ├── ClaudeCodeHarness
│   └── HermesHarness
└── Sandbox Backend
    ├── E2BBackend
    ├── DockerBackend
    └── LocalBackend

Hook Mechanism

Hooks provide extensible insertion points in the agent lifecycle (pre‑run, post‑run, error handling). Users can compose multiple hooks; the Runtime Manager executes them in the configured order, allowing custom logging, evaluation, or metric collection without modifying core logic.

# Create harness, sandbox, and hooks based on configuration
harness = create_harness(config.agent)
sandbox = create_sandbox(config.runtime)
hooks = create_hooks(config.hooks)

try:
    harness.setup(context, sandbox)
    hooks.before_run(context, sandbox)
    result = harness.run(context, sandbox)
    hooks.after_run(context, sandbox, result)
    return result
except Exception as error:
    hooks.on_error(context, sandbox, error)
    raise
finally:
    harness.cleanup(context, sandbox)
    sandbox.destroy(context)

7. Scalability and Extensibility

Gateway scaling : multiple Gateway processes handle different Sessions, distributing CPU‑intensive tokenization and trajectory management while preserving per‑Session consistency.

Runtime Manager scaling : when concurrent agents exceed ~1000, a single E2B SDK instance becomes a bottleneck; the solution is to run several Runtime Manager processes, each with its own SDK instance, and assign agents round‑robin.

8. References

Open‑source projects referenced in the implementation:

https://github.com/verl-project/verl

https://github.com/THUDM/slime

https://github.com/NVIDIA-NeMo/ProRL-Agent-Server

https://github.com/openclaw/openclaw

https://github.com/nousresearch/hermes-agent

https://github.com/openai/codex

https://github.com/shareAI-lab/learn-claude-cod