How Modern Data Lakes and AI Governance Transform Enterprise Analytics

Batch‑Stream Unified Processing with Apache Iceberg

The article describes a TC‑Iceberg extension that enables a lakehouse to handle both batch and streaming workloads without separate pipelines. The core architecture introduces two logical stores:

Base store – holds immutable snapshots of the full dataset.

Change store – records incremental inserts, updates, and deletes as separate files.

Read operations merge the base and change files on‑the‑fly (merge‑on‑read), while a background auto‑compaction job periodically rewrites change files into new base snapshots to limit read‑amplification. To reduce write‑amplification during merges, an automatic bucketing mechanism hashes the primary key of each record and assigns it to a bucket; merge tasks are then confined to a single bucket, dramatically improving distributed merge efficiency.

Future enhancements mentioned include sub‑second latency support for real‑time queries and native materialized‑view generation on top of the dual‑store layout.

AI‑Driven Governance for Game Data (Tencent Games)

A “governance‑as‑a‑service” model is built on Apache Gravitino. The solution defines a unified metadata model that captures table schemas, partitioning, and access policies across heterogeneous data sources. Data lineage is collected via the OpenLineage standard and enriched with Gravitino’s Facet extensions, which propagate custom metadata (e.g., business tags, sensitivity labels) through lineage events. This enables:

Cross‑engine lineage tracing (e.g., Spark → Flink → Presto).

Field‑level provenance for downstream analytics.

Automated policy enforcement based on unified metadata.

Unified Metadata and Lineage with Apache Gravitino

In multi‑cloud environments, enterprises face data silos and missing metadata. The article proposes using Gravitino as a central catalog that stores both technical metadata (schemas, partitions) and business metadata (ownership, tags). By integrating OpenLineage, the platform can automatically capture job‑level lineage and map it to Gravitino entities, achieving:

End‑to‑end lineage across heterogeneous storage systems (object stores, HDFS, relational databases).

Field‑level lineage for fine‑grained impact analysis.

Extensible Facet mechanism for custom attributes.

Reference implementation code is available in the public GitHub repository https://github.com/apache/gravitino.

Evolution of Xiaohongshu’s Multimodal Data Lake

Xiaohongshu migrated from a traditional Lambda architecture to an incremental compute model built on an Iceberg‑based lakehouse. Key techniques include:

Z‑Order sorting on frequently queried columns to co‑locate related rows.

Intelligent indexing that automatically creates secondary indexes for high‑cardinality attributes.

Incremental computation engine that processes only changed partitions, eliminating full‑recompute cycles.

Performance impact:

Query scan volume reduced by ~10×.

P90 query latency improved from >30 s to ≈5 s.

Benchmarks cover community feed and e‑commerce recommendation workloads, demonstrating the feasibility of real‑time analytics on the lakehouse.

Data+AI Multimodal Data Lake (Volcano Engine)

The solution combines three core components:

LAS AI – a library of ready‑to‑use data‑processing operators for text, image, and vector data.

LAS Ray – a heterogeneous compute scheduler that dispatches tasks to CPUs, GPUs, or specialized AI accelerators.

LAS Lance – a storage format that natively supports primary‑key indexes and vector indexes, enabling fast random access and similarity search.

Integration with ByteHouse provides a unified SQL layer capable of hybrid queries across structured tables and unstructured media stored in LAS Lance. Use cases highlighted:

Model pre‑training on large image/video corpora with sub‑second vector lookup.

Fine‑tuning pipelines that pull labeled examples via primary‑key joins.

Enterprise AI search combining keyword and semantic similarity.

Video mining where frame‑level vectors are indexed for rapid retrieval.

Technical details, including configuration snippets for LAS Ray job submission and LAS Lance schema definition, are available in the open‑source repository https://github.com/volcengine/las.