Understanding Multi‑Round Rollouts, Context Reconstruction, and RL Training in Agentic RL

Inference Pipeline

In a typical agentic inference step the call is split into three layers: the Agent harness internal state, the protocol‑specific LLM request, and the token sequence received by the model. The harness stores structured messages such as system, user, assistant, tool call, tool result, and environment observation. Before a model call the harness converts these messages into a protocol request (e.g., OpenAI Chat Completions, Anthropic Messages). The inference server then encodes the request, obtains the token IDs, generates output tokens autoregressively, and decodes them back to text. The protocol parser converts the text into a structured response that updates the harness state, possibly incorporating new environment observations or tool results.

Multi‑Round Rollout and Context Reconstruction

Extending a single call to multiple rounds introduces a trajectory \(\tau\) with message‑level records \(m_t, r_t, o_t\) for each step. Some harnesses store the session as an append‑only JSONL tree; compaction replaces early messages with summaries. Because the session is append‑only, the token input for later calls does not automatically preserve a strict prefix relationship. To maintain prefix consistency the protocol conversion must not modify existing messages, and system prompts, tool definitions, and the chat template must remain fixed.

In the Pi implementation, the session is saved as an append‑only JSONL tree. When restoring or compacting the session, Pi constructs the effective message list along the current branch, using summaries to replace older messages. Normal round‑by‑round calls use the current in‑memory messages, optionally applying a context transform before request generation, which still cannot guarantee prefix preservation.

RL Training Data Collection

A complete agentic rollout consists of model calls, harness state changes, tool/environment interactions, and the final reward. For training, each model call must record the exact input token IDs, the sampled output token IDs, and, if the training objective depends on the sampling probability, the log‑probability of each token under the rollout policy.

Training samples must accurately reconstruct the input tokens, output tokens, and a response/action mask that marks which tokens participate in loss computation. This requires a fixed tokenizer, chat template, and protocol conversion version.

When using a black‑box harness (e.g., Pi, Claude Code, OpenClaw), a serving gateway or proxy is placed between the harness and the inference service. The gateway translates protocol requests into inference calls, tags each request with rollout ID, attempt ID, and model version, and captures input_ids and, if possible, the token‑level log probabilities. The gateway also returns the structured response to the harness and writes the token‑level call record to a trajectory store for later training.

Cross‑Round Re‑Encoding and Trajectory Construction

In the next round the system does not simply concatenate the previous raw text. Instead it performs detokenization, output parsing, state update, protocol conversion, and input encoding. If the context‑preserving conditions hold and the serialization can be reduced to plain text concatenation, the token sequence may satisfy a strict prefix relationship: concat(text_i, text_{i+1}) ⇒ prefix(text_{i+1}) However, chat templates often insert control tokens at message boundaries, and detokenization may alter token boundaries, so the linear prefix condition is not generally guaranteed.

To keep each call usable as an independent training sample, the prefix condition must be enforced, or samples must be kept separate. Merging samples is possible only when token‑level conditions are satisfied; otherwise merging would change the conditioning sequence.

Trajectory Aggregation, Scoring, and Training Input

The raw token‑level call records are transformed through a pipeline:

token-level call records
    ↓
complete rollout records
    ↓
rollout groups
    ↓
verifier scores
    ↓
reward signals
    ↓
credit assignment
    ↓
training samples
    ↓
trainer

First, calls, environment traces, and version information are aggregated into a complete rollout record. Only when both model calls and environment records are present does the system proceed to scoring. Rollouts are grouped by predefined keys (task, initial state, sampling batch) to form rollout groups. Verifier scores—scalar, multi‑metric, or structured—are stored unchanged so that reward transformations can be audited later.

Reward transformation produces reward records while preserving a mapping back to the original verifier output. Credit assignment determines how each reward influences decisions within the rollout. The final sample constructor combines the token trajectory, group ID, rollout ID, strategy version, input/output tokens, response/action mask, and any required probabilities or weights.

Training systems consume these samples, compute the objective, and update model parameters. The design ensures that every generated token can be traced back to its originating call and conditioning sequence, and that rollouts can be independently scored, rewarded, and credited.

Each generated token must reference its originating model call and conditioning sequence.

Each rollout must reference its group, original score, training signal, and strategy version.

Reward transformations and credit assignment are explicitly configured; original verifier scores remain unchanged.

Trajectory splitting, merging, or packing does not alter rollout weights unless explicitly defined.