Can a Single AI‑Native Lakehouse Format Power Multimodal Data, Retrieval and Agent Memory?

At the 2026 AICon conference, Ma Jin, head of side‑memory at Volcano Engine, presented “Lance: From Multimodal Data Lake to Agent Lake,” describing the design and practice of the Lance lakehouse format.

Traditional data lakes evolved from Hive‑based pipelines to format‑centric systems such as Iceberg, Delta and Paimon. In the AI era, a single dataset must serve ETL/analytics, model training, vector and full‑text retrieval, sample replay and Agent memory, which raises new requirements.

Lance is positioned not as another vector database but as an AI‑native lakehouse format stack. Its architecture consists of a File Format layer (column pages, offsets, footer), a Table Format layer (fragments, versions, ACID), Index Formats (vector, full‑text, scalar), and a Catalog/Namespace layer for table discovery and cross‑engine integration.

This layered design pushes capabilities that previously required external systems—random point lookup, vector and full‑text search, versioning, branching, and multi‑engine collaboration—into the format itself, reducing external dependencies.

LanceDB exists alongside Lance; the former provides a lightweight, out‑of‑the‑box local vector and hybrid search engine, while Lance focuses on enterprise‑grade format and data management.

Since its open‑source launch in 2022, Lance has progressed alongside production deployments. Volcano Engine began contributing in 2024, targeting autonomous driving and embodied intelligence scenarios, and moved to productization and commercial rollout in 2025.

Why a New Format for Multimodal Data Lakes?

The speaker identified four AI‑driven demands:

Wide columns : multimodal tables may contain hundreds of columns, including images, point clouds, embeddings and scores, making traditional row‑group organization inefficient.

Low‑cost schema evolution : frequent addition or replacement of tags, embeddings or features should not require rewriting historic files; Lance’s fragment‑based organization allows new columns as separate DataFiles.

Mixed storage : instead of storing vectors in a separate vector DB and large objects in object storage, Lance converges structured fields, JSON, raw objects and vectors into a single queryable table.

Random point lookup : training shuffle, sample replay, and post‑retrieval re‑ranking need low‑latency reads of specific rows; Lance can achieve 1–2 I/O operations per random read, offering order‑of‑magnitude speedups over Parquet in certain cases.

File Format Change: Dropping Row Groups

Lance removes the Row Group concept from the file format and decouples data, metadata and footer. Row Groups, designed for HDFS, become sub‑optimal for object‑storage‑backed lakes, especially with highly variable column sizes and thousands of columns.

Instead, Lance reads the footer to locate column metadata, then uses that metadata to find the target row and data page. For large blobs, the Blob V2 layer provides lazy loading and streaming suitable for multimodal objects.

Relation to Parquet and Iceberg

Lance does not aim to replace Parquet or Iceberg. Parquet remains the de‑facto columnar file standard; Iceberg, Delta and Paimon excel at snapshots, transactions and multi‑engine governance for structured analytics. Lance complements them by providing AI‑specific capabilities such as random access, multimodal object handling, vector/full‑text indexing, frequent schema changes and cross‑scenario reuse.

Volcano Engine Practice

Since 2024, Volcano Engine has added features such as Branch, Transaction Commit Message, CDC, spatiotemporal types, Take API optimizations, FragmentSession API, distributed full‑text and vector indexes, JSON full‑text search, Java SDK, Spark/Ray/Daft connectors, LAS Catalog, datasets and heterogeneous storage. These enhancements form a complete multimodal data lake architecture: object storage and high‑performance file storage at the bottom, Lance format for data, compute engines for processing, and unified access for training, retrieval, evaluation and labeling.

Real‑World Scenario 1: Autonomous Driving Data Lake

Customers previously stored multimodal frames (≈20 MB each, hundreds of columns) in a KV store using Python pickle, with schema implicit in code, leading to high storage cost and difficult reuse. After adopting Lance, the data became a wide table; schema changes are expressed as new columns, and compression reduced raw data size to about 30 % of the original, handling a ten‑fold growth in volume.

Real‑World Scenario 2: Content Platform Data Cleaning and Scoring

Previously, each filtering, deduplication and scoring rule generated new Parquet intermediate files, causing fragmented tables, duplicate columns and high cost. With Lance, the workflow centers on a main table, branches and tags: cleaned data forms a processed table, branches add different scoring columns, and tags publish the final version. Incremental data consists mainly of new scoring columns, while historic data is fully reused.

Agent Memory Use Cases

Agent memory typically captures information from dialogues and tool calls, stores it as Markdown or in a vector DB, then performs splitting, embedding and indexing for later retrieval. LanceDB, being lightweight and ready‑to‑use, provides on‑disk vector and hybrid search without complex services, making it suitable for enterprise knowledge bases and personal document stores.

Two routes were highlighted: an enterprise‑oriented path using tags for millisecond backups and branches for team memory, with enhanced hybrid search and Chinese tokenization; and a personal‑oriented path leveraging Markdown editing strengths for semantic splitting, achieving a 30 % boost in memory capture and a 20 % increase in recall accuracy.

From Memory to Context

The discussion emphasized that memory is only a subset of the broader context asset, which includes documents, images, videos, code, decisions, tool call results, retrieval logs and cross‑session references. Managing this richer context requires a data base that can handle objects, versions, indexes and retrieval uniformly.

Conclusion

When data serves only reporting and offline analysis, traditional lake formats suffice. When data must simultaneously support model training, sample replay, RAG retrieval, multimodal object management and Agent context reuse, a new abstraction is needed. Lance answers this by turning the format layer into a common interface for AI data flow, governance and retrieval.