Skip to content
EROX Unified Data Plane Slashes Agent Cold Starts 50%: Read-to-Start, State Replicable
Cloud Native 31 min read

EROX Unified Data Plane Slashes Agent Cold Starts 50%: Read-to-Start, State Replicable

TencentOS EROX introduces a unified data plane that chains kernel direct mount, on-demand fetching, P2P distribution, incremental snapshots, and node-side cost reduction to accelerate AI Agent image startup and snapshot recovery, achieving 50GB workspace readiness in 1 second and 50%+ cold start reduction by enabling read-to-start and replicable state across nodes.

Tencent Architect
Tencent Architect
Tencent Architect

AI Agent Explosion: Sandboxes Stuck on Data Recovery

An AI Agent installs dependencies, runs tasks, and waits for large model responses — a single execution can last tens of minutes. When paused, the prepared environment and accumulated state must persist for the next run, possibly on a different node. Industry optimizations (millisecond cold start, resource pooling, hibernate/wake) target compute, but for stateful Agent and RL rollout workloads, fast startup does not mean the task can continue immediately. Beyond CPU scheduling and sandbox launch, the system must wait for image, workspace, or snapshot data to arrive.

Imagine 4,000 Agent sandboxes, each with a 50 GB workspace, migrating across nodes. If every startup requires pulling 50 GB from remote storage, wait time stretches from seconds to minutes. EROX focuses on this wait: accelerating image startup and snapshot distribution/recovery .

1. Problem: Stateful Loads Handle Data Far More Complex Than Images

1.1 Tasks Pause and Migrate, State Must Continue

Stateful loads demand four capabilities from the data system:

State persists across sessions and nodes. Subsequent executions depend on previously accumulated state. After a task ends or is interrupted, the next run must use that state on another node; data lifecycle cannot be bound to a single container.

Sleep during waits, fast recovery after. While waiting for model responses, compute may idle. The platform can save memory and disk state, destroy the instance to free resources, and later restore from snapshot.

Burst startup requires on-demand data arrival. RL rollout and evaluation launch thousands of sandboxes at once, a single job spawning tens of thousands. Data must arrive on demand, avoiding per-instance full preparation.

Diverse data shapes, distribution must be reusable. A sandbox includes container images, VM disks, code repos, toolchains, and memory snapshots; some images are used by only a few tasks. Building separate pipelines per data type increases storage cost and maintenance burden.

1.2 Both Runtime Environment and Task State Must Be Recoverable

Recovery requires preparing the runtime environment and task state together. The data plane manages four data types, often used simultaneously in one recovery:

Container image : read-only RootFS for container runtime.

VM disk : read-only VM RootFS or full disk for VM restore.

Workspace : read-write files and block data holding task-generated changes.

Runtime snapshot : memory, process, device, and other runtime state.

1.3 Concrete Example

Agent scenario requirements:

Business scale: 4,000 Agents , total data ~ 200 TB .

Per-Agent workspace: 50 GB , average run ~30 minutes.

Agents schedule across nodes; workspace must migrate with Agent.

Startup requirement: data directory mounted to Pod, start within 2 seconds .

Data requirement: runtime may install MySQL/npm/pip; changes must persist after task ends.

In this scenario, block storage per-Pod mount takes ~5–6 seconds; shared filesystems face random small-file write pressure from 4,000 Agents. The system must handle startup wait, runtime writes, and post-task state saving simultaneously.

1.4 Status Quo: Separate Pipelines, Industry Only Does Read Acceleration

Maintaining separate storage/distribution pipelines for the four data types causes three problems:

Artifacts cannot be reused : images, disks, snapshots use different pipelines; one piece of data requires multiple mechanisms.

Disks and snapshots still need full migration : images can lazy-load, but VM disks and snapshots are tens of GB; startup/migration must transfer the whole thing first — cold start in minutes.

Cross-node duplicate downloads, recovery may depend on source : repeated pulls increase transfer overhead and latency jitter; pulling directly from source ties recovery to source node/instance availability.

Stargz/SOCI enable container lazy-loading so containers start without full image download. Stateful tasks must also save runtime changes for next execution. EROX connects read, write, and save of all four data types into a single data plane, letting image startup and subsequent state recovery share the same infrastructure.

2. EROX Answer: One Unified Data Plane, Read·Write·Store Loop

To make this loop work, three questions must be answered: what format to store data, how to fetch on demand at startup, how to recover saved state on another node .

EROX unifies images, disks, workspaces, and runtime snapshots (read-only versions) into EROFS artifacts . EROFS separates metadata from data, suiting on-demand reads; different data sources thus reuse the same read, cache, and distribution capabilities. By default only one long-lived artifact is kept, avoiding the cost of storing both OCI images and EROFS artifacts just for image acceleration.

A recovery often needs multiple components. Capsule (胶囊) binds them with an immutable reference. It is essentially an Image Index (OCI image index) listing image, disk, runtime snapshot, workspace, etc. The business layer uses this reference to describe and copy components needed for recovery; the underlying on-demand read-only format remains EROFS.

EROX unified data plane architecture
EROX unified data plane architecture

Data handling in one task execution splits into three phases:

Read · Lazy Load : images and snapshots load on demand; startup does not wait for full data.

Write · Read-Write Separation : writable changes overlay the read-only layer; each snapshot is a version, traceable and rollback-able.

Store · Snapshot : during or at task end, changes solidify into a recoverable snapshot for next read.

Data arrives on demand, writes don't pollute read-only, changes solidify immediately — the three link end-to-end, forming a read-only → writable → read-only again loop.

3. Read · Lazy Load: Startup Without Waiting for Full Data

The read path must cut startup wait: get data needed to start first, fill the rest on access. When tens of thousands of instances start together, avoid concentrating all requests on remote storage.

EROX provides block-level and file-level on-demand reads. Block-level uses NBD (Network Block Device) or ublk (user-space block device) to attach EROFS block devices; file-level uses fanotify (file access notification) to populate data on access, no block device needed, finer granularity. Both share node cache.

Transport side can combine kTLS and io_uring to reduce data copy overhead; sequential reads can expand readahead windows to prefetch subsequent data.

Read path architecture
Read path architecture

3.1 Hotspot Prefetch: Get Startup Data First

On-demand loading reduces download volume, but scattered remote reads during startup still cause wait. EROX first uses eBPF to record representative loads, identifying files actually accessed at startup. At build time, hot file data is placed in a leading hot zone , metadata concentrated in a trailing metadata zone . At startup, these two contiguous ranges are read into node cache first, merging scattered reads into sequential transfers.

EROX only prefetches the startup working set; remaining cold data loads on access. Prefetching tens of GB whole images would still require full download and burden every node with full-image cache.

Result: relative to full pull, cold start reduced 50%+ ; for 10 GB+ large images, startup hotspot prefetch hit rate reaches 100%.

3.2 Cache and Direct Read: From Remote Fetch to Local Access

EROX fills node cache following the complete image's logical layout and records reliably persisted ranges. Local blocks are directly addressable by index and reusable; process restarts continue filling missing parts from already persisted data, avoiding re-download.

After the entire blob downloads and passes verification, it is sealed; subsequent mounts use the local EROFS image directly. Cache layout matches final artifact, so no post-download fragment reassembly is needed. Fully persisted segments switch to local direct read; unready parts continue on-demand fetch. The switch uses the same mount view — business need not remount — and gradually uses already-landed data.

3.3 P2P Distribution: Reduce Cross-Node Duplicate Origin Pulls

When multiple nodes read the same image or snapshot, duplicate requests hammer remote storage. EROX inserts P2P into the on-demand read path: on local cache miss, try peers first; only if peers lack the data, fall back to origin. Data already on one node becomes reusable by others, easing remote pressure.

EROX plans to embed P2P distribution inside the data plane process (in development), directly using existing cache, eliminating duplicate storage and inter-process copy from a separate P2P process. On peer miss, still fall back to origin; on-demand read and P2P no longer need separate deployments.

The same read path applies to snapshot distribution. When a sandbox hibernates and wakes, or needs cross-node restore, the target node reuses this cache and distribution chain to prepare VM disk snapshot and runtime snapshot data views, then the respective runtime completes restore.

4. Write · Read-Write Separation: Shared Read-Only, Private Writable

After environment starts, Agents install dependencies, modify code, generate files. EROX separates the shared read-only baseline from each instance's writable layer: existing data keeps being reused, each instance's changes stay isolated, later saved as snapshots.

Share read-only baseline, isolate instance changes. One parent snapshot serves multiple instances; each writes modifications to its own writable layer, unaffected by others. On the same node, these instances also share the read-only mount and its page cache, reducing duplicate footprint.

Save changes as versions, support rollback and branching. OverlayFS overlays read-only and writable layers; EROX builds writable changes into a new read-only snapshot, saves and publishes it. Historical snapshots stay read-only. If dependency install breaks or files are deleted, recover from retained history or branch from the same version.

Return writable quota on overwrite/delete. With eroxoverlay's dynamic quota, overwriting or deleting read-only data returns the masked portion's quota for reuse in the writable layer. This is quota accounting adjustment; shared read-only data remains intact.

Reduce large-file partial-write copy (experimental). Default uses native OverlayFS; experimental eroxoverlay copies only actually modified data blocks on supported write paths, cutting copy overhead for large-file partial updates.

Write path architecture
Write path architecture

From request perspective, read, write, publish follow separate paths:

A Cold Read : hit returns; miss falls back tier-by-tier — node cache → peer nodes → remote.

B Hot Write : goes straight to current writable layer; historical snapshots stay read-only; runtime hot writes never traverse remote read-only path.

C Publish : background solidifies changes into new snapshot; foreground business unblocked.

5. Store · Snapshot: Solidify Changes into Recovery Points

This task's changes remain in the instance's writable layer. To let another node use them, they must be saved as a snapshot. EROX supports high-frequency snapshots during execution: incremental save and cross-generation reuse cut per-snapshot storage and transfer; logical flattening controls read cost as versions accumulate.

Snapshot lifecycle
Snapshot lifecycle

5.1 Publish a Unified Cross-Node Restorable Reference

On snapshot save, EROX solidifies writable changes into a read-only artifact and uploads to remote (OCI registry, S3, etc.). The Capsule links image, disk, workspace, runtime snapshot, etc., so all data needed for one recovery is fetchable via a single version reference.

Publish by complete version : only after build, upload, and verification succeed does the new version become visible to restore; any incomplete step leaves the old version usable, avoiding partial-data reads.

At restore, the target node on-demand loads this version's read-only layers, creates its own writable layer, and continues subsequent changes. Since the snapshot is stored remotely, restore no longer depends on source instance survival .

5.2 Capture Stable View First, Then Build Snapshot

Online snapshots first capture a stable view, then complete build, upload, verification. Capture method and pause time depend on backend: Btrfs can take read-only subvolume snapshot while keeping the running mount; directory backend requires stopping related I/O, unmounting, building, and remounting. Application consistency still requires caller coordination.

After snapshot generation, the running instance continues using its original writable layer; only instances restored from the new version will use the new snapshot.

5.3 Incremental Save, Lower High-Frequency Snapshot Cost

To reduce frequent save and retry overhead, EROX uses:

Incremental snapshots : save only changes relative to parent, avoiding full state re-save each time.

Version logical chain : record parent-child version relations; unchanged data keeps referencing existing versions, letting multiple snapshots reuse the same data.

Idempotence : same content produces same result; duplicate submissions reuse directly, retries safe.

This mirrors Git's version management: parent-child relations track evolution, content-addressing reuses unchanged objects, same content yields same digest.

Large files in workspaces often change only small parts; incremental granularity must reach inside files. docker commit saves file-level changes as image layers; for tens of GB files, a partial change may still rewrite the whole file. EROX identifies changes at block level: new version saves modified blocks, unchanged ranges keep referencing parent data. Frequent version saves avoid re-uploading identical parts of large files; published snapshots remain restorable via on-demand path.

5.4 Flatten Read View, Avoid Per-Generation Lookup

High-frequency snapshots accumulate versions. EROX uses fsmerge to organize multi-generation data into a flat EROFS logical view: each inode's extents point directly to specific device and offset; unchanged blocks still come from parent. Version reuse remains, but reads no longer traverse the version chain layer by layer. The index is kernel-recognizable EROFS metadata; once data is ready, kernel direct-read path works.

EROFS 16-bit device index allows a single logical view to reference up to 65,535 extra data devices. If baseline uses one device and each generation adds one, the format supports up to 65,535 versions; multi-device layouts count against actual usage. This provides capacity headroom for continuous incremental snapshots; long-running systems still need version dependency management and data reclamation.

5.5 Switch Execution Node Only After Target Ready

After snapshot publish, the target node uses the on-demand read, cache, and P2P capabilities from Chapter 3 to prepare the restore data view.

Cross-node migration is coordinated by the control plane: source remains until target is ready as fallback; after target ready, switch ingress and reclaim source. Snapshot capture, publish, and restore can be organized in five steps:

Cut : control plane asks Source Agent to take a consistent snapshot point.

Publish : erox-snapshotter builds, uploads snapshot, publishes a RemoteReady (remotely pullable) new Capsule.

Restore : target node on-demand prepares view from new Capsule.

Ready : target ingress ready (AgentReady), source retained for rollback.

Switch & Exit : after target ready, control plane completes switch and source reclamation.

6. Node-Side Cost Reduction and Lifecycle Governance

As instances and versions grow, duplicate data, concurrent origin pulls, and stale mounts consume node resources. EROX controls persistent disk and memory overhead via data reuse, sparse storage, and reference reclamation:

Reuse identical data, merge duplicate requests : parent-child versions save only incremental diffs; same blocks reused across generations; multiple instances on same node share cache; in-flight duplicate requests merged into single origin pull; Capsule components referenced by content-addressing, identical components stored once.

Allocate space only for actually written regions : sparse snapshots allocate data space only for written ranges; unwritten regions never hit disk.

Reclaim by reference, release idle mounts : reference counting determines if data is still used; when reclamation conditions met, automatically clean unneeded data and release idle mounts, reducing resource accumulation in long runs.

7. Results and Data

7.1 VM Startup: Latency Cut 67%–90%

With recorded prefetch, Linux (GNOME), Windows, and macOS reach Ready in 7.72 s, 53.86 s, and 70.92 s respectively — reductions of 95.2%, 73.7%, and 93.5% versus remote full-download-then-start.

In on-demand startup tests, three images pulled 6.66%–19.08% of full image on first start. Windows second instance reused node cache, pulling 398.96 MiB vs 2.27 GiB — ~83% reduction.

VM startup latency comparison
VM startup latency comparison

These results show the working set needed for desktop readiness can be far smaller than the full image. Cache cuts data transfer, but OS initialization still takes time. Next, AI images with model loading and service initialization.

7.2 Hotspot Prefetch Further Cuts AI Large-Image Cold Start

Seven AI images tested across OverlayFS, Stargz, EROX on-demand, and EROX prefetch. Timing from image prep start to workload-defined business endpoint (API ready or task complete). EROX prefetch achieved lowest average latency in this set: vs OverlayFS reduced ~59%–95%, vs Stargz average reduced ~16.9%–56.5%. Hotspot prefetch overlaps critical data transfer with initialization, further cutting data wait during cold start.

AI image cold start comparison
AI image cold start comparison

7.3 50 GB Workspace, Second-Level Mount and Access

Back to the opening requirement: 4,000 Agents, each 50 GB workspace. Recovery passes "workspace mounted and accessible" then "application ready"; the business "start within 2 seconds after mount" targets the latter. Below measures the former.

In single 50 GB workspace restore test, EROX mount-and-access latency: same-region node 320 ms , cross-region overseas node 2049 ms . Target node first gets accessible data view; subsequent reads fill needed content on demand — cold data need not be fully moved before startup. The saved workspace thus becomes the foundation for the next execution.

50GB workspace mount latency
50GB workspace mount latency

Currently, EROX is deployed at scale in the company's Agent platform, RL training, and other scenarios, and continues evolving.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

AI Agentsnapshotlazy loadingP2PTencentOSEROFSincremental snapshotCapsuleEROXunified data plane
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.