How to Build Harness‑Native Agents Using OpenForge RL

OpenForge RL is a framework that decouples training from inference for large language model agents by inserting a lightweight proxy around the inference server and using a Kubernetes‑based orchestrator to run rollouts in remote containers. This eliminates the train‑deploy mismatch caused by stateful, multi‑process harnesses such as Claude Code, Codex, and OpenClaw.

1. Separate scaffolding for training and deployment

Modern agents run inside complex inference harnesses that manage multi‑turn interaction, tool calls, and context, making them stateful and difficult for open‑source SFT/RL stacks (veRL, slime) to express. The result is a “train‑deploy mismatch”. Additionally, harness rollouts require containerised environments that cannot share resources with training nodes, while most RL frameworks assume local rollouts.

2. OpenForge RL architecture: Proxy + Kubernetes

The core idea is to split training and inference using two lightweight components:

Lightweight Proxy : placed outside the inference server (e.g., vLLM), it intercepts every model call from the harness, forwards it to the RL framework’s inference engine, and records each prompt‑response pair as a trajectory. After rollout, the proxy aggregates terminal rewards and reconstructs the data into a standard RL sample format, allowing any algorithm (GRPO, PPO, REINFORCE) to be used without modifying the harness.

Kubernetes orchestrator (Orchard) : launches an isolated remote container for each rollout on Microsoft Azure, providing elastic scaling and support for arbitrary concurrent environments.

3. Automatic task‑environment synthesis pipeline

To address the scarcity of RL‑ready tasks for browser and desktop use, OpenForge RL adds a five‑stage pipeline built on the Claude Agent SDK (Opus 4.6):

Propose : generate candidate instructions from real‑world asset pools (ClawHub skills, ZClawBench, X Search API, AgentNet 22k commands, Synthetic‑Computer‑Use files) and store them in a shared SQLite for deduplication.

Prune : select the highest‑quality, most diverse N proposals.

Build : create a complete executable environment for each instruction, including tool servers, mock websites, mock data, Dockerfiles, and a verifier script.

Test : run another open‑source model (MiniMax‑M2.5 for Claw, Kimi‑K2.5 for Computer‑Use) to expose environment bugs and instruction ambiguities.

Refine : inspect trajectories and verifier scores, fix environments or rewrite instructions, and repeat until passing.

Cost‑wise, a filtered SFT task costs 5.2 minutes / $0.86, while an RL task costs 16.1 minutes / $4.36.

4. Experimental results

4.1 Claw Agent – everyday tool use

On ClawEval, QwenClawBench, and MCPAtlas, OpenForge‑trained models (≈30 B/3 B activation) outperform same‑size baselines (14.3 → 31.7 pass³) and Qwen3‑Coder, showing a clear advantage across all three benchmarks. Adding RL on top of SFT further lifts performance (pass³ 21.7 → 31.7, MCPAtlas 23.6 → 28.1), demonstrating the unique value of online interaction learning.

4.2 GUI Agent – 8 B model challenges larger models

An 8 B model reaches OSWorld‑Verified 37.7, close to the 235 B Qwen3‑VL score of 38.1, and Online‑Mind2Web 63.0 surpasses Kimi‑K2.5’s 60.4, effectively matching or exceeding models several times larger.

5. Three notable findings about Harnesses

Finding 1: Some harnesses are intrinsically hard to learn

When the same OpenForge‑Claw model is evaluated on four different harnesses, lightweight harnesses that allow custom tool registration (ReACT, ZeroClaw) learn best, while highly complex harnesses (OpenClaw) that expose tools via SKILL.md and have longer prompts show smaller SFT+RL gains. Simpler, tool‑aligned designs are therefore more performant.

Finding 2: Training on one harness transfers to unseen harnesses

Training solely on ZeroClaw improves performance on unseen OpenClaw and Codex by 3–5 points; mixing three harnesses yields the best overall gains, especially on the most complex harness (Codex +20.3). Exposure to diverse tool‑call syntax and control flow enhances robustness.

Finding 3: What does RL actually learn?

Comparing 100 SFT trajectories with 100 SFT+RL trajectories on ClawEval shows that RL reduces reliance on a generic shell (22.6 % → 13.9 %) and shifts toward specialized services, shortening trajectories slightly. RL mainly boosts agentic reliability—self‑verification, tool coverage, and multi‑step plan completion—while slightly reducing step efficiency. Error recovery remains the weakest ability (score 26), suggesting that terminal‑reward RL alone struggles to acquire robust failure‑handling.

6. Conclusion

The work demonstrates that harnesses can be elevated from deployment‑only engineering artifacts to first‑class research objects: “Train where you deploy” becomes a viable open‑source option.

OpenForge RL: Train Harness‑native Agents in Any Environment
https://arxiv.org/abs/2607.21557