How AReaL 2.0 Accelerates Self‑Evolving Agents

Problem Statement

Agents deployed in production generate large volumes of interaction trajectories—successful paths, failures, user corrections, tool results—but these logs are typically used only for diagnostics. Consequently, agents cannot improve on‑the‑fly despite running hundreds of self‑improving loops in real workloads.

Solution Overview (AReaL 2.0)

AReaL 2.0 extends the earlier AReaL v1.0 by moving the learning loop to the agent service side. It creates a closed‑loop where conversational interaction, trajectory collection, reward binding, and asynchronous training occur online without requiring developers to rewrite planning, tool, sandbox, or memory code.

Three Pillars of Self‑Evolving Agents

1. Agent Trajectory Data Protocol (ATDP) defines a step‑wise record that captures observation, internal state, chosen action, result, reward timing, model and tool versions, tenant, cost, permissions, and governance metadata. This transforms a complex task into accountable, replayable learning samples.

2. Enterprise‑grade Agentic Data Proxy implements the “how” of recording: it intercepts requests, sanitizes data, enforces permissions, persists trajectories, collects rewards, and manages replay, ensuring only qualified data reaches the training queue.

3. Agent Evolution Control Plane decides whether and where to apply updates (memory, tool routing, RL policy) based on statistics such as user correction rate, tool‑failure clusters, evaluator scores, cost signals, safety constraints, and distribution drift. It integrates offline regression tests, tenant‑level safety checks, canary releases, and version tracking.

Engineering Design

AReaL 2.0 decomposes training, inference, and weight‑update capabilities into independent, composable micro‑services linked by a “decouple‑then‑recombine” pattern:

Gateway – entry point for HTTP/WebSocket requests; forwards sessions to inference services and streams trajectory data to the training pipeline.

Router – maintains session affinity across backend workers to preserve multi‑turn context.

Data Proxy – stores session history, assembles AgentRequest objects, and provides training data to downstream workers.

Agent‑Compute Worker – executes agent logic, collects incremental events, invokes LLM backends (e.g., SGLang, vLLM) for inference, and runs large‑scale training kernels (Megatron, FSDP).

Controller – orchestrates component lifecycle, scaling, health checks, and runtime management.

These components together enable a full pipeline from online request ingestion, session persistence, trajectory capture, to training‑time updates.

Practical Demonstrations

Hermes Agent demonstrates a low‑intrusion integration: swapping the standard inference backend with the AReaL Agent‑Compute Worker brings real interactions into the RL loop without modifying planning, tool, or memory modules. Repository: https://github.com/areal-project/AReaL/tree/main/examples/hermes

Claude Code Agent provides an end‑to‑end software‑engineering pipeline. It filters training samples, rewrites issue descriptions for clarity, runs millions of sandboxed environments on distributed NPU clusters, and applies KPop stabilization with token‑level adaptive filtering to avoid RL‑induced crashes. After roughly 800 training steps the model shows a stable score increase. Repository: https://github.com/areal-project/AReaL/tree/main/examples/swe

Results

Both examples report that after ~800 online training steps the model’s evaluation score rises consistently, confirming that real‑world trajectories can be turned into effective RL training data.

References

Paper: “Next‑Generation Agentic Reinforcement Learning Systems Enable Self‑Evolving Agents” – https://arxiv.org/pdf/2607.01120

Project homepage: https://github.com/areal-project/AReaL