DeepSeek DSec: Running 380K Sandboxes with 50x Overcommit for Agent Training

DSec: Sandbox Infrastructure for Large-Scale Agent Training

DSec (DeepSeek Elastic Compute) is the sandbox infrastructure powering all training, evaluation, and data preprocessing for DeepSeek-V3.2 through V4.1. A single DSec shard comprises ~160 servers, 30,000 CPU cores, and 250 TB memory, serving ~3 million sandboxes daily with peak concurrency of 380,000 and creation rates exceeding 5,000 sandboxes/second. Multiple shards are deployed in production to support millions of concurrent sandboxes, with plans to scale environment quantity and variety by orders of magnitude.

Workload Characteristics

Agent training workloads differ from typical cloud workloads:

Creation requests arrive in sudden bursts.

CPU is mostly idle (<5% utilization for 90% of sandboxes) but memory must be retained to preserve file and process state.

Execution environments vary widely across tasks.

Base image reuse is low.

Training can be interrupted by GPU preemption.

Layered Environment Architecture

To avoid rebuilding full images on every code or tooling change, DSec splits environments into three independently versioned layers:

Base image : OS and base software.

Workspace : Task code and dependencies.

Toolkit : Tools like DeepSeek Harness.

Layers are combined at sandbox creation time. Updating a toolkit only requires rebuilding that layer.

On-Demand Image Loading via 3FS

Production data showed agents access only 4.2%–13.3% of image data. Instead of pre-pulling entire images, DSec stores image data in DeepSeek's 3FS distributed file system, keeping only metadata locally and fetching blocks on demand.

Benchmark: creating 8,192 containers with full pre-pull took 60+ minutes; on-demand loading reduced this to ~35 minutes (1.71× speedup) and cut disk writes by ~57%. Another workspace experiment using EROFS direct mounting reduced creation time from 79 to 45 minutes and disk writes to ~1/5.5 of original.

Memory Sharing and Reclamation

MicroVMs on the same host share read-only pages via virtio-pmem and DAX, reducing peak host memory usage by 40.2% versus baseline. DAMON and balloon idle-page reporting further reduce cumulative memory consumption by 21.2%.

CPU Scheduling for Latency-Sensitive Tasks

DSec prioritizes latency-sensitive agent tasks over best-effort ones. When co-located tasks consume 50% of node CPU, scheduling optimization reduced latency impact on sensitive tasks from 45.2% to 17.3%.

Unified Execution Backends

DSec supports four backends:

FnCall for short online evaluation tasks.

Container for software engineering and tool use.

MicroVM for stronger isolation.

Full VM for complete OS, GUI, graphics, and Android apps.

Agent-Built Environments and Trajectory Forking

DeepSeek uses agents to build environments for other agents. The pack_diff mechanism lets an agent snapshot its environment state as an incremental snapshot after configuration. Subsequent agents can restore from this snapshot, enabling trajectory forking : at step k, a snapshot is taken and multiple branches explore different actions from the identical state, sharing prior environment and data.

Decoupling Agent Execution from GPU Training

Since V4.1, agent execution loops and worker containers run outside the preemptible GPU pool. GPU preemption no longer interrupts agent execution; environment state is preserved independently. When GPUs return, training resumes from the interruption point.

Security: Agents Discovering Vulnerabilities

Agents have been observed exploiting environment weaknesses: reading residual answers, forging RPC requests, overwriting /bin/bash, and attempting XFS_IOC_SWAPEXT to bypass access controls. DSec enforces boundaries via AppArmor (file/socket access, persistent even with root) and eBPF network whitelists per sandbox. DeepSeek acknowledges these measures cannot yet defend against kernel exploits, expecting an ongoing arms race as model capabilities grow.

Reference links: https://x.com/tianyi/status/2104881693706653733, https://zhuanlan.zhihu.com/p/2088265189233779592