Cutting 60% of Agentic RL Execution Costs with Alibaba Cloud FC Sandbox

1. What Execution Environments Agentic RL Really Needs

Agentic RL moves from generating answers to completing tasks in an environment, so each Action changes the Environment and the next Observation depends on those changes. A rollout can involve dozens or hundreds of steps, and the Verifier must check the final state left by the Agent. Therefore the environment is no longer a passive resource; it directly affects correctness, stability, and efficiency.

2. OSWorld as a Representative Workload

OSWorld requires the Agent to act on a full desktop using Screenshot, Accessibility Tree, Mouse, Keyboard, Command and File operations. A single trial must keep state continuous and keep Observation and Action aligned, while a full evaluation must isolate, reuse and schedule hundreds of independent trials.

3. Four Core Requirements for a GUI Agent Environment

State changes caused by the Agent must be retained until the Verifier finishes.

Observation and Action must form a closed loop around the same state.

Repeatedly created environments must share a stable public baseline.

Each trial’s state must be isolated from others.

These guarantees ensure trial‑level semantic correctness and, at scale, test whether a single baseline can be repeatedly instantiated and scheduled under limited concurrency.

4. Harbor Evaluation Flow

Harbor organizes a evaluation as a Trial consisting of Environment Setup, Agent execution, Verifier scoring and Result collection. The tests/ directory is injected only after the Agent finishes, preventing the Agent from seeing the scoring logic. The same Sandbox instance is used for both Agent and Verifier so that the Verifier observes the exact state left by the Agent.

Key point: GUI benchmarks need state continuity, not just identical configurations.

5. GUI Environment Guarantees

The environment must keep a bidirectional data path: Observation → Environment, Action → Environment. All APIs (Command, File, Screenshot, Accessibility Tree, Mouse, Keyboard) must point to the same state space. For example, a file written by a Command must be readable by a desktop application, and a mouse click must appear in the next Screenshot.

6. Baseline vs. Trial State

The baseline contains OS, systemd, X server, GNOME desktop, common applications (Chrome, VS Code, etc.), envd, OSWorld service and shared verifier dependencies. The trial state holds task‑specific material, solution files, and any changes the Agent makes (files, desktop layout, processes). This separation avoids cross‑trial contamination while allowing fast trial startup.

7. Adapter Layer

The Environment Adapter maps Harbor’s BaseEnvironment methods ( start(), stop()) to the sandbox lifecycle and maps exec(), file upload/download to the sandbox data plane. It also translates Verifier exit codes and connection errors into distinct failure categories (task failure vs. infrastructure failure).

8. Control Plane vs. Data Plane

Control Plane manages sandbox instance lifecycle (create, get, pause, resume, kill). Data Plane handles commands, file access and GUI interactions inside a running instance. This separation reflects the fact that lifecycle calls are rare, while GUI interactions occur hundreds of times per trial.

9. Scaling Experiments

Full‑scale OSWorld evaluation : 361 tasks, 100‑step configuration, Qwen3.6‑Plus model, 50 concurrency, 4 vCPU / 8 GiB sandbox. Two runs showed 361/361 and 360/361 trials completed scoring, mean rewards of 0.383 and 0.397, and only one trial exception in Run B.

Reproducibility test : Created new sandboxes from the same template at concurrency levels 1, 10, 20. P90 create time stayed around 1.2 s, and P90 create→Desktop‑Ready time stayed near 18 s, showing no significant slowdown with concurrency.

Large‑scale rollout : 369 OSWorld tasks, 15 steps, 30 concurrency, same sandbox spec. Wall‑clock time 2 h 25 m 52 s, total sandbox‑hours 68.34, mean trial lifetime 667 s, concurrency fill rate 93.7 %. 356 trials completed scoring; 13 failed with VerifierTimeoutError.

10. Cost Analysis

Compute‑only cost for the sandbox (4 vCPU / 8 GiB) is ¥0.48 / h. Multiplying by 68.34 h gives ¥32.80 per run, i.e., ¥0.089 per trial. Compared with Azure ML F4s v2 public price (~¥1.14 / h), the FC sandbox is about 57.9 % cheaper per hour and roughly 61.6 % cheaper for the whole job.

11. Conclusion

The Harbor + FC sandbox integration turns a GUI environment into a scalable infrastructure that satisfies the strict state‑continuity requirements of Agentic RL. It supports full OSWorld evaluation, reproducible trial creation, and high‑concurrency rollout while cutting compute costs by around 60 %.