How Paimon and Milvus Build an AI‑Native Multimodal Data Lake

As AI applications evolve from model capability to Retrieval‑Augmented Generation (RAG) and finally to continuously running agents, the underlying data infrastructure must also change. The author argues that the traditional "dual‑system" architecture— a data lake for raw and batch data plus a separate vector database for online retrieval—creates structural burdens that become acute when agents require frequent reads, writes, and versioned data.

Why AI‑Native Multimodal Data Lakes Are Needed

In production, teams typically store raw data, features, training sets, and batch results in a lake while keeping embeddings and ANN indexes in a vector database. This leads to two "truths" that must be kept in sync, causing duplicated storage, costly index rebuilds, consistency windows, and poor elasticity.

Four Problems of the Dual‑System Architecture

Duplicate storage : embeddings are stored both in the lake and the vector DB, inflating storage costs and complicating data ownership.

Index redundancy and rebuild : indexes live outside the lake’s versioning and lineage system; any change to source data or embedding models forces a full index rebuild.

Consistency window : updates to the lake take time to become visible to the online service, making it hard to guarantee that online queries see the same data as offline analysis.

Lack of elasticity : online services must keep compute resources alive even for cold historical data that is rarely accessed.

Evolution of Agentic Workloads

The author describes three stages that illustrate how data access patterns change:

Static RAG : documents are chunked, embeddings are generated once, and a single retrieval is performed.

Tool Use : models start invoking databases, APIs, and workflows within a session, adding a layer of action.

Production Agent : agents run continuously, repeatedly read and write data, and maintain long‑term context, turning data into a live material for both online serving and offline analytics.

These stages highlight that the problem is no longer "is vector retrieval fast enough" but whether the data foundation can support continuous read/write, version evolution, and online‑offline collaboration.

Desired Features of an AI‑Native Multimodal Data Lake

The target architecture should provide four capabilities:

Unified storage : vectors, scalars, blobs, and indexes coexist as first‑class citizens with governance and versioning.

Streaming‑plus‑batch : the same data can be written in real‑time, batch‑backfilled, and processed offline without maintaining two separate pipelines.

One data, many engines : the lake data can be served by low‑latency retrieval engines and also scanned by Spark, Flink, etc., without being tied to a specific engine.

Elastic compute : resources can be provisioned on‑demand for online services while low‑frequency historical data can be kept cheap.

Both Databricks (Lakehouse) and Zilliz (vector DB) are pursuing similar directions; the Paimon × Milvus integration represents an open‑source path that combines a lake format with a high‑performance vector engine.

Paimon and Milvus: Roles and Collaboration

Paimon acts as the lake foundation, handling storage, governance, and evolution. It introduces a native VECTOR<FLOAT, n> type, a dedicated Vortex storage path for vector columns, and a Global Index that can cover B‑Tree, Bitmap, Vector, Full‑Text, and hybrid indexes. This makes vectors first‑class data that can be versioned and governed alongside scalars and blobs.

Milvus provides the retrieval and service layer, offering ANN, hybrid search, scalar filtering, distributed scaling, and production‑grade low‑latency serving. Its key innovation for this integration is the External Collection feature, which allows Milvus to query data directly from an external lake format without copying the primary data.

The combined architecture consists of:

Flink streaming, CDC, and batch back‑fill writing to a unified Paimon multimodal table.

Paimon tables storing vectors, scalars, blobs, and Global Index metadata with versioning.

Milvus External Collection reading the same lake snapshots to provide ANN, hybrid search, and filter capabilities.

This solves three core problems: eliminating duplicate data movement, enabling computation on a single data source, and maintaining service freshness through snapshot‑based refresh.

Three‑Layer Readiness Model

To assess whether the lake and vector engine truly share the same data, the author proposes a three‑layer readiness check:

Semantic layer : alignment of schema, types, and column semantics (e.g., Paimon VECTOR<FLOAT, n> ↔ Milvus FLOAT_VECTOR).

Physical layer : both sides can read the data in its native open format without copying, even if internal storage layouts differ.

Retrieval layer : clear contracts for index construction, ANN parameters, query semantics, and result‑to‑row mapping.

Currently Paimon and Milvus have achieved readiness at the semantic and physical layers; the retrieval layer still requires further work on index reuse and query alignment.

Retrieval Layer Evolution

The retrieval component should progress in two phases:

Phase 1 – Serving Index : Milvus builds and manages the index for high QPS, low‑latency online services, immediately eliminating the duplicate data problem.

Phase 2 – Open Index : Paimon’s Global Index becomes a shared, reusable index that multiple engines can consume, supporting online‑offline co‑evolution.

Both paths share the same underlying lake data; the main difference is who owns the index and whether it can be reused across engines.

Real‑World Example: Autonomous Driving Data Loop

In an autonomous‑driving scenario, petabyte‑scale sensor data, images, video, and embeddings must support both real‑time similarity search for online decision making and offline analysis for corner‑case discovery, de‑duplication, and model retraining. Using the dual‑system would exacerbate the four problems (duplicate storage, massive index rebuilds, inconsistent views, and costly cold‑data residency). The unified vector lake lets Flink continuously ingest sensor streams into a Paimon multimodal table, Milvus serve low‑latency similarity queries on the same snapshots, and Spark/Flink batch jobs perform offline mining and model updates—all on the same factual data.

From Open‑Source to Enterprise Productization

For production use, enterprises need catalog, permission, lineage, operations, elasticity, and SLA guarantees. Alibaba Cloud’s solution combines DLF (One Data Catalog, lake storage management, governance) with a managed Milvus service offering two index paths: a serving index for high‑QPS workloads and an open index for multi‑engine sharing. The roadmap includes three practical steps:

Establish the One Data foundation : use Paimon multimodal tables with DLF catalog and governance.

Integrate the service layer : add Milvus, selecting the appropriate index path based on workload.

Close the data loop : bring in Flink streaming, batch recompute, data quality tasks, and agent workflows to achieve continuous discovery, processing, write‑back, and serving.

This incremental approach lets organizations first unify data ownership, then add low‑latency serving, and finally achieve a full Agentic Data Loop where online and offline processes operate on the same governed data.

Conclusion

The author summarizes five key takeaways: (1) dual‑system architectures hit structural limits for agents and multimodal workloads; (2) Paimon brings vectors into the lake as native objects with versioned governance; (3) Milvus’s External Collection moves from copying data to serving directly from the lake; (4) the pragmatic path is the "One Data First" strategy that first shares primary data and then evolves open indexes; (5) this open‑source route avoids lock‑in to a single engine and delivers clearer data ownership.

Ultimately, the decisive factor is not which engine is fastest, but where the data lives, who owns it, and whether it can evolve without being locked into a proprietary system.