CUA-Sandbox: 6.2x Agent RL Throughput by Sharing Runtime, Isolating Only Mutable State

Problem: Environment Provision Becomes the Bottleneck in Agentic RL

Computer-use agents (CUA) interact with web pages, applications, and operating systems through the same interfaces as humans. Training these agents increasingly relies on reinforcement learning, requiring massive numbers of rollouts in real, executable software environments. Recent work like DeepSeek's DSec system scaled sandbox concurrency to hundreds of thousands at the cluster level, but inside each sandbox the traditional approach still allocates a full container or VM per environment. This means memory and lifecycle overhead grow linearly with the number of environments, limiting the experience that can be collected under a fixed resource budget.

Key Insight: Only Mutable State Needs Isolation

The research team observed that application code and initialized services can be shared across rollouts; only the mutable state — files, sessions, databases, application state — must be forked per environment. They ask: must an independent computer-use environment own a full execution runtime?

CUA-Sandbox Architecture

The system separates three planes:

Execution Plane: Browser, web services, databases, desktop runtimes reside here and are shared across all logical environments.

State Plane: Each logical environment holds a private state capsule . A binding table maps environment ID to the correct capsule, routing actions and producing observations and rewards.

Control Plane: Lifecycle operations — reset, snapshot, clone, fork — execute as transactions following a freeze → drain → stage → atomic publish protocol.

State Capsule Design

Mutable resources are classified by how they must be saved:

Authoritative state (databases, user-created files): stored as copy-on-write (CoW) private deltas on top of an immutable base.

Derived state (caches, search indexes): kept in private namespaces; rebuilt on demand when cheaper than copying.

Ephemeral state (temporary IPC endpoints, display resources): recreated when the capsule activates.

Each capsule carries an environment ID, a generation number, and a manifest recording component versions and dependencies. The manifest commits only when all required components are ready, ensuring database branches, file overlays, and application profiles advance together as a single generation.

Execution by State

Every action carries an environment ID. The control plane resolves the current binding and acquires a generation lease. For authoritative resources, reads see a combined view of base data plus the environment's private branch; writes go only to the private branch. Derived services and temporary interfaces use per-environment namespaces and endpoints. All requests, callbacks, and background tasks inherit the same context, so observations and evaluators read the same generation state.

Three conditions must hold for safe runtime sharing: complete resource bindings, disjoint writable state across environments, and single-generation consistency per operation. An execution context is marked ready only after every contract resource is bound. Binding granularity varies by application: web services can select private state per request, while browsers and desktop interactions require session-level profiles, configs, display, and message-bus endpoints.

Transactional Lifecycle Operations

To avoid mixing generations during in-flight requests, state transitions follow:

Freeze routing, reject new requests.

Drain in-flight requests.

Stage the successor capsule.

Atomically publish the new generation number and manifest.

Generation leases guarantee ongoing work stays bound to the old version; expired requests after the switch are rejected.

Runtime Reuse: N Logical Environments on Q Physical Slots

Because state and runtime are decoupled, the number of materialized capsules need not equal the number of active execution contexts. The system maintains logical environments and allocates physical execution slots on demand. Inactive environments persist only as capsules. Rebinding a slot involves quiescing the current context, mounting the target capsule, recreating ephemeral resources, updating routing, and passing a readiness check. N logical environments can therefore reuse Q physical slots. Resources that cannot be safely shared fall back to private backends, leaving original software interfaces and evaluators unchanged.

Experimental Results

Experiments on WebArena, VisualWebArena, and OSWorld used official Docker environments as baselines. Three experiment types: paired inference, matched agent post-training, and system overhead measurement.

Inference: Success Rates Unchanged or Improved

Across 4 models (GPT-5.6-Terra, Qwen3.5-9B, Muse-Glimmer-30B, Gemma-4-31B-IT) and 3 benchmarks (12 comparisons), CUA-Sandbox success rates were never lower than Docker. Example: GPT-5.6-Terra on OSWorld rose from 67.29% to 70.62% .

Post-Training: SFT and RL Gains Preserved

Using Qwen3.8-27B as teacher and Qwen3.5-4B as student, both backends produced nearly identical RL improvements (WebArena: +3.08 vs +2.95 pp; OSWorld: +3.87 vs +3.88 pp). Final scores matched or slightly exceeded Docker.

Efficiency: Throughput, Memory, Storage All Improved

At 8 concurrent environments:

Throughput up 2.22–6.20x: VisualWebArena environment prep time dropped from 10.404s to 0.504s ; batch time from 44.762s to 7.215s ; throughput from 0.179 to 1.109 envs/s .

Memory up to 9.2x saved: OSWorld per-environment memory from 2,653.8 MiB to 837.3 MiB .

Storage up to 504x saved: Web environment from 6,503.70 MiB to 12.90 MiB .

Per-site breakdown: VisualWebArena Shopping create accelerated 15.17x , fork accelerated 17.84x ; Reddit storage saved 9,535x ; GitLab memory saved 51.2x .

Ablation Studies

Copy-on-Write: On a 3.6 GB Shopping capsule, removing CoW increased

create from 0.071s to 5.476s (<strong>77.1x</strong>), <code>checkpoint from 0.082s to 4.695s (<strong>57.3x</strong>), <code>fork from 0.054s to 4.772s (<strong>88.4x</strong>), and storage delta from 0.11 MiB to <strong>3.67 GiB</strong>. CoW mainly eliminates state materialization overhead.</code></code>

Non-Database State Management: On WebArena task 21, adding non-DB state increased prep time by 0.844s and reset by 1.086s, with 4 HTTP readiness probes returning 200 — the measurable cost of covering resources beyond databases.

Transactional Publishing: In 100 stress tests, naive publishing caused mixed-generation execution and in-flight request publishing every time; the full protocol yielded 0 occurrences of both issues and rejected all 100 requests issued during the freeze phase. State reuse requires coordinated state transitions.

Scaling to 16 Concurrent Environments

At 16 environments, WebArena throughput: CUA-Sandbox 0.324 envs/s vs Docker 0.132 envs/s ; per-environment memory: 436.1 MiB vs 2,082.6 MiB . VisualWebArena throughput: 0.732 envs/s vs 0.208 envs/s . As concurrency grew from 4 to 16, CUA-Sandbox per-environment memory stayed stable (WebArena 492.9 → 436.1 MiB), showing runtime-sharing savings are not diluted by higher concurrency.

Resource Model and Trade-offs

The paper provides a resource model explaining savings:

Memory: from shared runtime (avoiding repeated initialization) and from keeping inactive environments as capsules only, without occupying execution slots.

Storage: from sharing initialized writable state, not duplicating read-only images.

Lifecycle operations: avoid rebuilding the full runtime but still pay for quiescing, state preparation, rebinding, and readiness checks; latency depends on backend and private state size.

An appendix ablation with 8 logical environments shows: official Docker runtime memory 15,560 MiB; adding CoW alone (separate runtimes) reduces to 13,882 MiB; adding runtime sharing brings it down to 3,261 MiB . Thus memory savings come mainly from runtime sharing, while CoW mainly reduces storage and lifecycle overhead (writable storage from 52.99 GiB to 23.19 GiB).

Implications for Agentic RL Scaling

The authors conclude that when forking an environment approaches the cost of copying a state, previously impractical training paradigms become feasible: forking multiple trajectories from arbitrary intermediate states, tree-search-style rollouts, and low-cost rollback and retry. Cluster-level scheduling (like DSec) solves "can we afford enough environments?"; state-level isolation determines "how cheap can an environment get?" Combining both forms the true scaling foundation for agentic RL.