From Data Lake to State Lake: Redesigning Storage Infrastructure for the Agent Era

AI Workload Evolution and Storage Paradigm Shift

Recent years have seen AI applications evolve from simple model calls to conversational assistants and now to autonomous agents capable of tool use, task execution, environment interaction, and long‑term memory. This shift creates three distinct data workload phases:

Training : massive datasets (hundreds of PiB–EiB), high‑throughput loading and checkpoint writes at TB/s scale.

Inference : ultra‑low latency, high concurrency, KV‑cache‑driven workloads where millisecond or microsecond I/O delays directly affect end‑user performance.

Agent : heterogeneous, high‑load, observable, and self‑evolving workloads that require flexible isolation, rapid sandbox creation, and extensive trace logging.

These workload changes drive a storage paradigm evolution from traditional Content Storage —focused on massive, persistent, cold‑archived objects—to State Storage , which emphasizes short‑lived tensors, millisecond‑level availability, and low‑cost branching.

Three Capability Directions of Storage Agent Infra

Volcano Engine’s Storage Agent Infra is organized around three capabilities:

Sandbox Store : Provides root‑fs mounting, snapshots, fast start‑up, pause/resume, and state preservation for agent sandboxes.

Artifact Store : Handles persistent, shared, and distributable agent inputs, outputs, and task artifacts.

Agent Observation & Evaluation : Collects logs, traces, sessions, and evaluation data to close the optimization loop.

Sandbox Store Details

Sandbox Store must satisfy fast start‑up, state saving, controllable capacity, clear isolation, flexible recovery, and robust communication. The solution combines:

EBS : High‑frequency snapshots, ultra‑long snapshot chains, and instant cloning enable agents to save and roll back to any step, supporting parallel exploration.

EFS : Provides POSIX‑compatible shared storage with configurable bandwidth, burst capability up to 2 TB/s, and up to 30 million IOPS for small‑file intensive workloads.

MQ LiteTopic : Assigns an isolated message topic per agent session, guaranteeing strict ordering and supporting millions of concurrent topics for multi‑agent collaboration.

These layers together address sandbox state, shared data, and communication needs.

Artifact Store Details

Artifact Store serves the data flow of agents. Two paths are defined:

POSIX‑heavy workloads : EFS delivers strong file semantics, high concurrency, and burst bandwidth, with sub‑millisecond latency and up to 3000 million IOPS, reducing task start‑up and artifact generation time.

Object‑storage‑centric workloads : Volcano Engine TOS offers virtually unlimited bucket capacity, elastic scaling, and tiered storage (standard, infrequent, archive) to store multimodal outputs cost‑effectively. Data flow features automatically tier cold data to lower‑cost storage while keeping it accessible.

Additional features such as SenseFlow (multimodal understanding) and ContextBucket (persistent agent memory and workspace) extend artifact capabilities beyond simple file storage.

Agent Observation & Evaluation

Because agent outcomes cannot be judged by fixed rules, the system records execution traces, sessions, and metrics. TLS AgentLoop ingests these observations, provides dashboards, and integrates with Cozeloop for evaluation datasets, enabling developers and platform teams to compare agent versions, diagnose bottlenecks, and continuously improve task quality.

Real‑World Scenarios

Three concrete scenarios illustrate how the three pillars combine:

R&D testing agents : Continuous snapshotting via high‑frequency EBS snapshots enables version rollback and branch switching during debugging.

Multimodal creation agents : Input assets are pre‑processed and stored, while outputs are persisted in an Agent Bucket on TOS for public access and distribution.

Intelligent office agents : User files reside in a personal space, collaborative agent content lives in a shared “Claw” space, and ADrive’s dual‑space architecture provides seamless collaboration and AI‑powered search.

These cases stress the need for instant sandbox start‑up, high‑throughput artifact handling, and persistent, searchable agent memory.

Re‑thinking Storage for the Agent Era

Agent workloads reshape storage from a passive data lake to an active state lake that supports rapid environment provisioning, state preservation, and continuous observability. Success metrics shift from raw capacity and bandwidth to the ability to sustain the entire agent lifecycle—seconds‑level environment launch, millisecond‑level state access, and intelligent, traceable execution.

Volcano Engine’s Storage Agent Infra, built on EBS, EFS, TOS, ADrive, TLS, and MQ, re‑assembles these components into a cohesive “state base” that not only stores data but also understands tasks, retains context, and participates in the optimization loop, ultimately becoming an “agentic teammate” rather than a mere storage backend.