Orchard: Microsoft’s Open‑Source Agent Framework Hits 0.28 s Latency with 1,000 Sandboxes

Problem

Training or evaluating AI agents requires isolated sandboxes for each command, file operation, or test run. Reinforcement‑learning rollouts need hundreds to thousands of sandboxes in parallel, making environment cost dominant.

Orchard Architecture

Orchard Env is a Kubernetes‑native sandbox orchestrator that separates the control plane from the hot path. The Kubernetes API server handles only low‑frequency pod creation, deletion, and network‑policy management. High‑frequency operations (command execution, file I/O, health checks) are sent directly to an HTTP agent running inside each sandbox, bypassing the API server.

All sandbox pods share a single namespace sandbox-pods protected by a deny‑all‑egress NetworkPolicy, reducing API calls per sandbox lifecycle.

Horizontal scaling is achieved by running multiple orchestrator replicas that share sandbox state via Redis; a distributed lock guarantees serialized commands per sandbox.

Layered Design

Orchard Environment : Kubernetes‑native sandbox service + Python SDK; can launch thousands of isolated containers on demand.

Research Layer : Open‑source training methods Orchard‑SWE, Orchard‑GUI, Orchard‑Claw and downstream projects OpenWebRL, OpenForgeRL.

Training Layer : Vendored fork of the slime RL stack (upstream THUDM/slime) located under examples/orchard/ as a git submodule pointing to MSR-Orchard/slime.

Core Capabilities

Control‑plane / hot‑path separation : Measured average command latency of 0.28 s for 1,000 concurrent sandboxes (README performance table).

Arbitrary base images : An init container injects a self‑contained CPython 3.11 runtime, libc, and dependencies, so the user image does not need Python.

Built‑in harnesses : codex, claude, pi, opencode, hermes are pre‑installed and appended to PATH without overriding existing binaries.

Sync and async SDK : SandboxClient (synchronous) and AsyncSandboxClient share the same REST contract, provide context‑manager semantics, automatic retries, and graceful cleanup on SIGINT / SIGTERM.

Network isolation : Calico NetworkPolicy denies all outbound traffic by default; outbound access must be explicitly allowed per sandbox.

Multi‑replica scaling : Redis shares sandbox records and task status; a per‑sandbox execution lock is upgraded to a distributed lock, ensuring strict command ordering across replicas.

Three‑fold leak protection : TTL reclamation (default 2 h), heartbeat timeout (180 s), pending‑creation timeout, plus a 5‑minute cluster reconciliation loop that removes orphan pods.

REST API contract : SDK is a thin wrapper around a set of REST endpoints, making the service language‑agnostic.

REST API Endpoints

GET /

– service banner GET /health – health check (no auth) GET /resources – cluster resource summary POST /sandboxes – create sandbox GET /sandboxes/{id} – query sandbox GET /sandboxes/{id}/wait – block until ready DELETE /sandboxes/{id} – delete sandbox POST /sandboxes/{id}/heartbeat – refresh heartbeat POST /sandboxes/{id}/exec – execute command (optional wait=true) WS /sandboxes/{id}/exec/pty – interactive PTY via WebSocket POST /sandboxes/{id}/apply_patch – apply git patch POST /sandboxes/{id}/files – upload file (base64) GET /sandboxes/{id}/files – download file (base64) GET /sandboxes/{id}/files/list – list directory GET /jobs/{job_id} – query async task result GET /jobs/{job_id}/wait – block until task finishes

SDK Methods

exec

– run a command, returns

JobResult(stdout)

apply_patch

– write a diff and execute

git apply

upload_file / upload_content

– upload a local file or raw bytes download_file / download_content – retrieve file bytes list_files – list directory contents start_heartbeat / stop_heartbeat – control heartbeat thread (default every 60 s) get_job – fetch async task result check_env – probe sandbox environment delete – remove sandbox

Configuration Options (selected)

max_concurrent_execs=400

– global execution concurrency limit max_concurrent_creates=50 – per‑replica sandbox creation limit k8s_api_concurrency=100 – limit on concurrent K8s API calls agent_port=9090 – port on which the sandbox agent listens exec_connect_retry_window=45s – retry window for agent connection failures sandbox_ttl_hours=2 – maximum sandbox lifetime heartbeat_timeout_seconds=180 – heartbeat timeout before sandbox is considered dead default_block_network=True – deny all egress by default use_redis=True – enable shared state across orchestrator replicas

Key Implementation Details

Any Docker image with glibc 2.17 (CentOS 7) to glibc 2.39 (Ubuntu 24.04) has been validated, including SWE‑bench images.

Harness binaries are appended to PATH without overwriting existing tools.

API keys are injected at runtime via the env parameter; no credentials are baked into images.

Harness payloads are mounted read‑only at /opt/sandbox-tools; version information is available via cat /opt/sandbox-tools/VERSIONS.

Three harnesses ( Copilot CLI, Cursor Agent, Gemini CLI) are omitted because they require GLIBC_2.28, which is not universally available.

Usage Example (synchronous)

from orchard_env import SandboxClient

with SandboxClient() as client:
    with client.create_sandbox("python:3.11-slim") as sandbox:
        result = sandbox.exec("echo 'Hello, Orchard!'")
        print(result.stdout)

Running a harness inside an arbitrary image:

with client.create_sandbox(image="ubuntu:22.04") as sandbox:
    result = sandbox.exec(
        "codex exec 'summarize this repo'",
        env={"OPENAI_API_KEY": "sk-..."},
        timeout=600,
    )

Overall Call Chain

SDK issues a request → orchestrator receives it.

Orchestrator checks a local cache for the sandbox agent; if missing, it falls back to a K8s lookup.

Agent client connects directly to the sandbox pod IP (e.g., pod:9090/exec) via an aiohttp connection pool.

Task state is written to a job store: queued → running → succeeded/failed, queryable via GET /jobs/{id}.

When the context manager exits or a termination signal arrives, cleanup code runs ( atexit, SIGINT / SIGTERM) and the three‑fold reclamation (TTL, heartbeat, pending) removes the sandbox.

Design Decisions

Control‑plane separation : Only pod creation/deletion and network‑policy management go through the Kubernetes API server; command execution and file I/O go directly to the sandbox agent, which explains the 0.28 s latency at 1,000 concurrent sandboxes.

Shared namespace : All sandbox pods run in the sandbox-pods namespace with a deny‑all‑egress policy, eliminating per‑sandbox namespace churn.

System Layers

User application layer: training loops, evaluation scripts, any language client.

Client SDK: SandboxClient / AsyncSandboxClient with context management, retries, and cleanup.

Orchestrator: FastAPI service, horizontally scalable, manages lifecycle, scheduling, task state, Pod watching, Kubernetes client, Agent client, and Redis.

Kubernetes cluster: Two node pools – sys runs orchestrator and Redis; sbx runs sandbox pods with label workload=sandbox and taint, auto‑scaled.

Sandbox Pod Structure

Init container agent-injector : Copies a self‑contained agent (CPython 3.11, libc, FastAPI, Uvicorn) into a shared volume.

Main container : Runs the user image, mounts the same volume, starts the agent via --library-path without touching the image’s own libc, Python, or PATH.

Agent endpoints : /health, /exec, /files/upload, /files/download, /files/list, /exec/pty (WebSocket); reachable only within the cluster.

Repository

Project repository: https://github.com/microsoft/Orchard Upstream training stack:

https://github.com/THUDM/slime