Deep Dive: Multimodal Data Lake Formats – Paimon vs. Hudi vs. Iceberg

Background

Managing multimodal data (images, audio, video) in a data lake introduces three core challenges: (1) huge volume variance from kilobyte records to gigabyte files, (2) sparse access where training reads only fragments or columns, causing inefficient I/O with traditional columnar formats, and (3) the need to fuse scalar SQL filtering with vector ANN retrieval in a single query.

Positioning and Comparison

From the 2025‑2026 evolution, the three communities converged on multimodal solutions with distinct approaches:

Paimon embeds a vector store, object storage, and data lake directly into the table.

Hudi reuses its mature indexing subsystem to handle vectors.

Iceberg leaves vector handling to external partners (e.g., Lance, Milvus) while strengthening its structured and semi‑structured foundation.

Paimon

Paimon 1.4 marks a strategic upgrade from a real‑time lake to an AI‑native multimodal lake. Its key pillars are:

Column‑separated architecture with a global Row ID : each row receives a unique, immutable ID at write time; files store rows with consecutive IDs, enabling precise row location and automatic cross‑file association. Adding a new column (e.g., user interest tags) only writes a new file with the column and its Row IDs, reducing storage cost.

BLOB data type : stores large unstructured objects separately from structured columns. Queries on structured fields skip BLOB files entirely. The BLOB field is defined with BYTES / BINARY / BLOB, and the system records the external path, offset, and length so Spark/Flink can stream the data on demand, avoiding OOM.

Unified global index (Lumina) and B‑Tree index :

Lumina vector index (DiskANN) builds an ANN index for ARRAY<FLOAT> columns, supporting semantic search, image retrieval, recommendation, and RAG. Each table supports a single vector column; NULL vectors are excluded.

B‑Tree global index provides high‑performance scalar lookups with equality and range predicates, automatically applied by Flink SQL or Spark SQL.

Pre‑filter query flow : scalar index first narrows candidates, then vector search runs. Example query:

SELECT id, title FROM vector_search('my_db.image_embeddings', 'embedding', array(0.9F, 0.1F, 0.0F, 0.4F), 10);

Engineering enhancements : Deletion Vector + placeholder logical deletion for safe compaction, and Blob Compaction to merge small files.

Hudi

Hudi introduced multimodal support in version 1.2, leveraging its plug‑in table‑format framework.

Native VECTOR and BLOB types : Embeddings, images, and videos coexist with structured columns in a single table. The official "Unstructured Data Quick Start" demonstrates an end‑to‑end workflow where a VECTOR column stores image embeddings and a BLOB column stores raw image bytes, enabling a single SQL statement to perform top‑K similarity search and retrieve the original image:

-- Declare VECTOR(1024) for embedding, BLOB for raw image bytes
-- hudi_vector_search performs ANN, read_blob fetches the image
SELECT id,
       hudi_vector_search(embedding, :query_vec) AS score,
       read_blob(image_bytes) AS image
FROM image_table
ORDER BY score
LIMIT 5;

Metadata Table as unified index carrier : All indexes reside in a single Merge‑On‑Read (MOR) table under .hoodie/metadata/ using HFile for efficient key lookup.

ACID consistency : Indexes stay synchronized with data tables, supporting asynchronous index building without exposing partial writes.

Multiple co‑existing indexes : Data Skipping (files, column_stats, partition_stats), precise record lookup (record_index, secondary_index), and extensible SQL CREATE INDEX for bloom filters or expression indexes.

Vector index integration : The ANN index is a first‑class index type stored in the Metadata Table, inheriting Hudi's transactional, concurrent, and incremental update guarantees.

Iceberg

Iceberg adopts a different strategy: it does not embed a vector index but solidifies its semi‑structured data, governance, and row‑level operation efficiency via Spec v3. Vector and unstructured retrieval are delegated to external engines such as Lance, Milvus, or compute frameworks like Daft.

In practice, Iceberg often pairs with Lance/LanceDB: Iceberg handles structured analysis and SQL, while Lance provides random access to multimodal data for training. On platforms like Alibaba Cloud DLF 3.0 and Volcano Engine LAS, Iceberg is cataloged alongside Paimon and Lance, with Milvus/PAI supplying vector search.

Iceberg does not offer out‑of‑the‑box lake‑internal vector search; users must integrate external components. Its strengths lie in an open ecosystem, multi‑engine access, and strong governance/compliance.

Conclusion

The following recommendation matrix summarizes the three frameworks:

Choose Paimon for real‑time ingestion, built‑in lake‑internal vector search, and unified handling of large AI‑generated objects.

Choose Hudi if you already have a Hudi stack, need strong upsert/CDC capabilities, and want to extend to multimodal data.

Choose Iceberg when you prioritize a multi‑engine, cloud‑agnostic foundation with robust governance and primarily semi‑structured data.

For extreme multimodal training performance (random access to massive samples), combine Iceberg + Lance .