How NVIDIA’s Polar Enables Any Agent Framework to Plug Into Reinforcement Learning

Ignored Engineering Challenge

Current AI‑Agent reinforcement‑learning training requires tightly coupling the agent harness with the training pipeline to capture the token sequence generated at each step. Switching to a different harness (e.g., Claude Code, Codex, Qwen Code) forces a complete rewrite of the integration code, including harness adaptation, API formatting, KV‑cache handling, and token alignment, leading to massive engineering overhead and fragile pipelines.

Polar’s Core Idea: API Proxy + Black‑Box Execution

Polar treats the agent harness as a black box and inserts an API‑proxy layer between the harness and the LLM inference service. The proxy intercepts all requests and responses, records the full prompt and generation tokens, and later reconstructs a trajectory usable for RL training without modifying the harness code.

Architecture Breakdown: Three Key Components

First Layer – Asynchronous Gateway

The gateway splits the rollout lifecycle into four stages (runtime initialization, ready buffering, harness execution, post‑processing) and assigns each stage to an independent worker pool, allowing non‑blocking parallelism. A practical optimization is runtime pre‑warming, where environments are prepared in advance to avoid repeated container startup and dependency installation.

Second Layer – Model API Proxy

The proxy records every request to the LLM API, capturing complete prompt and generation tokens. For streaming requests it concatenates the full response before logging. In multi‑agent scenarios it maintains separate interaction logs for each agent session.

Third Layer – Trajectory Reconstructor

This component rebuilds training trajectories from the recorded interactions. Two reconstruction strategies are offered:

Per‑request : processes each rollout independently without context merging; simple but generates redundant prefixes and yields low GPU utilization.

Prefix‑merging : merges common prefixes across adjacent rounds, keeping only incremental tokens; improves GPU utilization but must handle token mis‑alignment caused by context compaction.

Experiments show that the prefix‑merging strategy achieves noticeably higher GPU utilization and throughput.

Experimental Results: Surprising Gains on a 4B Model

Evaluations were performed on the SWE‑Bench Verified benchmark using a Qwen3.5‑4B base model and the GRPO RL algorithm. The following improvements were observed after training with Polar:

Codex harness: from 2.6% to 25.2% (+22.6 points)

Claude Code harness: from 19.4% to 24.2% (+4.8 points)

Qwen Code harness: from 25.6% to 26.2% (+0.6 points)

Pi harness: from 0.0% to 6.2% (+6.2 points)

The Codex harness showed the most dramatic increase, nearly ten‑fold, indicating that tool‑use capabilities (shell execution, file editing) provide substantial upside when combined with RL training.

Key Observations

1. Harness Choice Matters More Than Expected The same model and RL algorithm yielded performance differences of over 20 percentage points across different harnesses, suggesting that the agent scaffolding itself is a critical, often underestimated factor.

2. Offline Data Generation Is Viable Polar can also generate offline trajectories by running a strong model on a harness, recording the interactions, and then using the data for distillation or offline RL, which is cost‑effective for expensive harnesses.

3. Integration With NVIDIA’s NeMo Ecosystem Polar is registered as an environment in NeMo‑Gym and connects with the NeMo‑Aligner training framework, making it accessible to teams with NVIDIA GPU clusters.

Comparison With Prior Work

Before Polar, Agent RL followed two main paths: (1) modifying rollout frameworks (e.g., AgentGym, Agent‑R) to embed agent logic directly, incurring high maintenance when switching harnesses; and (2) using SFT with rejection sampling, which avoids true RL but loses exploration and credit‑assignment benefits. Polar introduces a third path—injecting a proxy layer without touching either the harness or the training framework—achieving clean engineering decoupling.

Limitations

Only supports text‑in‑text‑out API calls; multimodal inputs (images, audio) are not handled.

Experiments are limited to 4B models; performance on larger models remains unverified.

The asynchronous architecture adds communication overhead, requiring sufficient parallelism to amortize the cost.

Conclusion

Polar solves a practical engineering problem by enabling any agent framework to connect to RL training without modifying harness code, using an API proxy and trajectory reconstruction. The most valuable insight for researchers is the large variance in RL outcomes across different harnesses, highlighting the importance of agent scaffolding design. For engineering teams, Polar’s decoupled architecture offers a reusable pattern where rollout services are exposed as a consumable API, allowing independent evolution of the agent harness and the training pipeline.