A Comprehensive Overview of the Latest Advances in Agentic Reinforcement Learning

Search-o1: Agentic Search‑Enhanced Large Reasoning Models

During inference the model emits the special token templates <|begin_search_query|> and <|end_search_query|> to delimit a search query. When the end token is seen the LLM pauses, extracts the query, calls a retrieval function, and inserts the retrieved documents wrapped by <|begin_search_result|> and <|end_search_result|>. The reasoning chain then continues with the external knowledge.

The Reason‑in‑Documents module runs in parallel to the main chain. It analyses the retrieved documents together with the current query and previous reasoning steps, generates an intermediate reasoning sequence, extracts refined knowledge, and injects that knowledge back into the main chain. This avoids token explosion and reduces noise from raw documents.

Search‑R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Search‑R1 introduces multi‑turn retrieval using the token templates <|search|> / <|/search|> for the search action, <|information|> / <|/information|> for the retrieved content, <|think|> / <|/think|> for internal reasoning, and <|answer|> / <|/answer|> for the final answer.

Training experiments compare PPO and GRPO. GRPO converges faster but its performance drops later; PPO converges more slowly yet remains stable and achieves higher final performance. Rewards are binary based on answer correctness.

The paper proposes Retrieved Token Masking : tokens inside <|information|> and <|/information|> are excluded from loss computation, preventing the model from being confused by external content.

ToRL: Scaling Tool‑Integrated RL

ToRL adds a Python execution tool to the search‑only paradigm. During a rollout, when the termination marker ```output is detected, text generation pauses, the extracted code block is executed, and the observation is inserted back into the context using the format:

```output
OBSERVATION
```

The observation is then used for subsequent reasoning until the model produces a final answer or more code.

Key tricks:

Tool Call Frequency Control: a hyper‑parameter C limits the maximum number of tool calls per response; calls beyond the limit are ignored, forcing pure text reasoning.

Error Message Processing: only the last line of a stack trace (e.g., "NameError: name 'a' is not defined") is kept to reduce context length.

Sandbox Output Masking: execution results are excluded from loss, analogous to Retrieved Token Masking.

Reward design gives +1 for a correct answer, –1 for an incorrect answer, and an additional –0.5 penalty for code‑execution errors.

ToolRL: Reward Is All Tool Learning Needs

Experiments show that longer trajectories do not necessarily improve performance; overly long reward sequences can hurt.

Finer‑grained reward decomposition yields more stable and effective learning, and dynamic reward scaling (curriculum‑like) helps the model transition from simple to complex behaviors.

Rewards consist of:

Format Reward: checks for the presence and correct order of special tokens such as <|think|>, <|tool_call|>, <|response|>.

Correctness Reward: evaluates tool‑name matching, parameter‑name matching, and parameter‑value matching. The final answer reward is largely omitted; learning is driven by format and tool‑correctness rewards.

Acting Less Is Reasoning More! Teaching Model to Act Efficiently

The work defines a tool productivity metric (correct answers / tool calls) and designs a reward that penalises unnecessary calls while still rewarding correct answers. When the model makes m tool calls but the estimated minimum is n, the tool‑efficiency reward decays with a cosine/sine function, reaching its maximum when m = n and decreasing otherwise.

R1‑Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

A two‑stage Agentic RL framework:

Stage 1 – Encourage Retrieval

Retrieve Reward: any rollout that calls the external retrieval tool at least once receives a reward, encouraging proactive search.

Format Reward: enforces a structured output. The thinking process must be wrapped in <|think|>…<|/think|>, the final answer in <|answer|>…<|/answer|>, and the search query in <|begin_of_query|>…<|/end_of_query|>. Queries cannot be bypassed.

The stage‑1 reward is the sum of Retrieve Reward and Format Reward, teaching the full "think → retrieve → format" pipeline.

Stage 2 – Use Retrieval Effectively

Reward signals shift to Answer Reward plus Format Reward. The model must still call the retriever but now must produce an accurate answer based on the retrieved content. The total reward remains the sum of the two components.

Group‑in‑Group Policy Optimization (GiGPO) for LLM Agent Training

GiGPO addresses sparse, delayed rewards by introducing a two‑level advantage estimation:

Macro level (trajectory‑level): group complete trajectories and compute a "macro relative advantage" to compare overall performance.

Micro level (step‑level): identify "anchor states" that appear across different trajectories; actions taken in the same anchor state are grouped, and a "micro relative advantage" is computed only for those actions, providing fine‑grained credit assignment.

The macro and micro advantages are combined (e.g., weighted sum) to guide training without extra critic models or additional rollouts.

ARTIST: Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning

The model outputs a structured template in the order: <|think|>…<|/think|> (internal reasoning) <|tool_name|>…<|/tool_name|> (tool or environment call) <|output|>…<|/output|> (tool response) <|answer|>…<|/answer|> (final answer)

Rollouts consist of these fragments; at each step the policy decides whether to continue thinking, invoke a tool, or adjust based on tool output.

Reward components:

Answer Reward: positive reward for a correct answer inside <|answer|>.

Format Reward: encourages the exact sequence and structure of the template.

Tool Execution Reward: measures tool‑success rate (successful calls / total calls) to promote correct tool queries.

Agent RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving

Experiments reveal three trends as training steps increase:

Code‑usage frequency rises.

Average response length grows.

Task accuracy on math problems improves.

The authors term this the "Agent RL Scaling Law": tool‑use frequency, response length, and accuracy positively correlate with training scale.

Replay‑buffer filtering improves stability and efficiency:

Groups with accuracy > 0.8 are discarded (gradient too small).

Groups with accuracy < 0.2 are discarded (likely noise).

Only groups with accuracy in 0.2–0.8 are retained for training.

A maximum tool‑call limit N_max is enforced; if a rollout exceeds it, a message "Tool call limit reached" is inserted and the model must finish using only internal reasoning.

ARPO: Agentic Reinforced Policy Optimization

Entropy spikes after each tool use, indicating high uncertainty from external feedback. ARPO allocates more exploration to these high‑entropy steps by combining global trajectory sampling with entropy‑driven sub‑sampling: when entropy change exceeds a threshold, branch sampling is triggered. Branching stops when the sub‑sampling budget is exhausted or all paths terminate.

Advantage attribution splits a rollout into:

Shared Path: tokens before the branch point are identical across rollouts and share the same advantage.

Branched Path: tokens after the branch diverge; each path receives its own advantage based on its outcome.

The reward aggregates answer correctness, format correctness, and multi‑tool collaboration.

Chain‑of‑Agents (CoA): End‑to‑End Agent Foundation Models via Multi‑Agent Distillation and Agentic RL

CoA uses a single LLM to simulate a multi‑agent collaboration workflow. First, multi‑agent knowledge distillation generates trajectories that include role‑playing, tool calls, and observations. These trajectories are used for supervised fine‑tuning (SFT). Then RL optimises the model on verified agentic tasks.

Reward design adds a binary "LLM‑as‑Judge" signal for web‑agent tasks: a large model (Qwen‑2.5‑72B‑Instruct) judges semantic match with the reference answer, yielding 0 or 1 reward, replacing traditional F1/EM metrics that fail in open‑domain settings.

SFT provides a cold‑start for collaborative reasoning; RL refines tool usage, role switching, chained thinking, verification, and self‑reflection.

ASearcher: Beyond Ten Turns – Unlocking Long‑Horizon Agentic Search with Large‑Scale Asynchronous RL

Typical online RL training uses fewer than ten interaction turns, leading to low‑quality data (misunderstanding, inaccurate web extraction, misleading answers). ASearcher contributes two ideas:

Creates a harder dataset by injecting relevant context for correctness while deliberately obfuscating the question to increase difficulty.

Builds a large‑scale asynchronous training engine that allows trajectories with dozens of tool calls and long token outputs, dramatically improving training efficiency for long‑horizon search.

An additional summarisation step over web content mirrors the Reason‑in‑Documents module from Search‑R1.

Atom‑Searcher: Enhancing Agentic Deep Research via Fine‑Grained Atomic Thought Reward

Previous works use sparse outcome‑based rewards, providing little signal for intermediate reasoning, and the generic <|think|> token lacks guidance for sub‑steps.

Two core ideas are proposed:

Atomic Thought paradigm: split a large <|think|> block into multiple explicit sub‑units (e.g., <|Plan|>, <|Reflection|>, <|Verification|>) to make reasoning more structured and supervisable.

Process + Result mixed reward with dynamic scheduling: a Reasoning Reward Model (RRM) scores Atomic Thoughts, yielding an Atomic Thought Reward (ATR). Early training emphasizes ATR (process) while later stages shift weight toward the final answer reward (outcome).

The approach follows an SFT‑then‑RL pipeline to handle the increased template complexity.

Deep Think with Confidence

The paper proposes confidence‑based metrics to differentiate high‑ and low‑quality reasoning paths:

Bottom‑10% Group Confidence: compute token‑level confidence over sliding windows, take the lowest 10 % windows, and average them to represent the weakest segment of a trajectory.

Lowest Group Confidence: select the single lowest‑confidence window as the metric for the entire trajectory.

Two usage scenarios are explored:

Online thinking: monitor confidence during generation; low‑confidence paths can be aborted, filtered, or adjusted, and final voting weights are biased toward high‑confidence trajectories.

Offline thinking: generate multiple complete trajectories, then select or weight them based on confidence scores before deciding the final answer.

The method is training‑free and operates at inference time, similar to self‑evolution works such as SE‑Agent.

Towards General Agentic Intelligence via Environment Scaling

Scaling environments, rather than merely increasing model size or data, is identified as a key bottleneck. Tools are abstracted as underlying database read/write operations (e.g., get_weather(city) → Read, book_flight(city) → Write). Tools are grouped by domain; each domain is represented by a database schema. Automatic construction of tool sequences, initial database states, and user intents yields diverse, automatically generated environments, reducing manual scenario design.

Training proceeds in two stages:

Stage 1: train agents on general domains to acquire basic tool‑calling skills.

Stage 2: fine‑tune agents on domain‑specific contexts for higher fidelity performance.