How 1565PB of High-Quality Datasets Are Reshaping Lakehouse Architecture for AI

Background: Explosive Growth of High-Quality Datasets

2026 is designated as the "Year of Data Element Value Release." China's National Data Administration reports 120,000 high-quality datasets totaling 1,565 PB as of June 2026, with quarterly growth exceeding 60% (up from 960 PB in Q1). The "Implementation Plan for Promoting Industry High-Quality Dataset Construction" (April 2026) outlines six special actions across 19 key domains. Seven pilot cities (Chengdu, Shenyang, Hefei, Changsha, Haikou, Baoding, Datong) have completed 119 PB of annotated data, employing 140,000 annotators serving 425 LLM projects and driving 26.3 billion RMB in related industry value.

Core Characteristics of High-Quality Datasets

2.1 Exponential Scale: Storage Scalability & Cost Pressure

Datasets have grown from TB to PB scale with accelerating growth. Traditional MPP coupled storage-compute architectures suffer from low resource utilization (CPU <20%) and physical cluster limits. Compute-storage separation is mandatory, but requires solving metadata management, caching, data tiering, and compute pushdown.

2.2 Multi-Modal Data: From Table-Centric to Unified Association

Medical imaging datasets exemplify the challenge: a single sample may contain DICOM images, JSON annotation contours, doctor diagnosis text, and pre-computed feature vectors — all requiring strong correlation and traceability. Traditional lakehouses store unstructured data as file paths in VARCHAR columns, lacking native management. Required capabilities: unified Row ID association, native Blob storage with lazy loading, vector indexing, and cross-modal joint retrieval.

2.3 Annotation Iteration: Fine-Grained Version Management

Annotations undergo continuous iteration — multiple rounds of labeling, review, correction — producing many versions. Beyond table-level time travel, the architecture must support row/sample-level version tracing to answer questions like "What changed in this sample's annotation between v1.2 and v1.3?" or "Which samples were labeled by Annotator A and reviewed by QA B?"

2.4 AI Training Access Patterns: Random, High-Concurrency, Low-Latency

Microsoft's study of 400 real deep learning jobs found ~46% of low GPU utilization stems from data operations. AI training exhibits random access + small files + high concurrency + low latency, unlike BI's sequential scans. Data loading, storage access, and small-file handling inefficiencies cause expensive GPU idle time.

Challenge 1: Storage Architecture

3.1 Core Contradiction at Massive Scale

Coupled architectures bind storage and compute costs; scaling for capacity leaves compute idle. Industry data shows compute-storage separation reduces overall storage cost 40-60% and total infrastructure cost 50-70%.

3.2 Four-Layer Compute-Storage Separation Design (Paimon + StarRocks)

Unified Storage Layer: Parquet/ORC on object storage (S3/OSS/HDFS) with open table formats (Paimon/Iceberg/Hudi) providing ACID, schema evolution, snapshot management. Fully decoupled, multi-engine sharing.

Distributed Compute Layer: Stateless compute nodes (StarRocks BE, Flink, Spark) fetch metadata via Catalog, read directly from object storage. Elastic scaling, multi-cluster physical isolation (write, query, ETL clusters).

Multi-Level Cache System: Three tiers — Local File Cache (hot data files), OS Page Cache, Engine Internal Page Cache. Full cache hit matches coupled performance; partial hit ~10% degradation; cold ~30% degradation.

Metadata Service Layer: Manages schema, partitions, snapshots, manifests. StarRocks 3.2 added Metadata Cache for partitions/manifests; 3.3 introduced Distributed Plan for parallel manifest parsing, scaling linearly with compute resources.

3.3 Four-Tier Hot/Cold Tiering & Cost Optimization

Data exhibits strong hot/cold patterns. A four-tier system automatically demotes data via lifecycle policies:

Hot (SSD, <7 days)

Warm (OSS Standard, 7-90 days)

Cold (OSS Infrequent Access, 90-365 days)

Archive (OSS Archive, >365 days)

For 1 PB: all-SSD costs ~1.23M RMB/month; tiered mix (8% hot, 22% warm, 50% cold, 20% archive) costs ~159K RMB/month — an 87% reduction.

-- Paimon table lifecycle policy
ALTER TABLE quality_dataset SET TBLPROPERTIES (
  'hot_retention_days' = '7',
  'warm_retention_days' = '90',
  'cold_retention_days' = '365',
  'archive.enable' = 'true',
  'compaction.strategy' = 'time-series',
  'file.compaction.level' = '5'
);
-- StarRocks tiered storage policy
ALTER TABLE dataset_metrics SET STORAGE POLICY = 'tiered'
PROPERTIES (
  'hot_partition_count' = '7',
  'cooldown_time' = '2592000',
  'storage_medium' = 'SSD'
);

3.4 Lake Format Selection & Recommendation

Comparison of Apache Paimon, Iceberg, Hudi, Delta Lake. For high-quality dataset scenarios, Paimon + StarRocks is the most competitive combination:

Strong real-time updates: LSM-tree architecture, native primary key UPSERT/DELETE, annotation updates visible in seconds.

Partial updates: Field-level incremental updates avoid full row rewrites for annotation completion or quality score updates.

Multi-modal native evolution: Paimon 2.0 multi-modal tables use Global Row ID to unify vector indexes, object files, scalar columns, full-text indexes in one table.

StarRocks deep optimization: Native C++ Reader replaces JNI, delivering >5x MOR table read throughput and >80% QPS improvement in high-concurrency point queries.

Challenge 2: Data Governance & Quality Control

4.1 Metadata Management: From Technical to Full-Dimensional

Governance complexity exceeds traditional warehouses. An image classification dataset may carry dozens of attributes: collection time, device, lighting, categories, annotator ID, review status, quality score, copyright type. Three metadata categories:

Technical: Schema, partitions, formats, locations — auto-collected from lake formats.

Business: Meaning, source, annotation specs, quality standards, use cases, owners — manually maintained.

Operational: Access frequency, users, downstream dependencies, quality scores, issues — auto-generated via instrumentation.

Open-source options: DataHub (30+ connectors, push/pull, large-scale heterogeneous) and OpenMetadata (lightweight, clean API, medium teams).

4.2 Data Lineage: From Table-Level to Operator-Level

Lineage enables precise impact analysis: when annotation rules change, which downstream datasets and training jobs are affected? When model performance drops, is it data or model? Traditional tools have three limitations: low precision (<80% SQL parsing accuracy), missing row-level granularity (false alerts), static analysis only (no runtime flow).

Operator-level lineage breaks these: AST-based full SQL parsing >99% accuracy, handles nested queries, dynamic SQL, stored procedures. Row-level pruning via WHERE clause analysis reduces manual assessment nodes by 80%+.

Storage selection:

<500K nodes: MySQL adjacency list, <100ms for 3-hop queries.

Million-node scale: NebulaGraph distributed graph DB, 6-hop from 8s to 200ms.

Middle ground: DataHub uses Elasticsearch + RDBMS with application-layer BFS.

Best practice: embed lineage in CI/CD for "preventive control" — auto-verify schema change impacts on reports, metrics, AI models, data contracts; block or require approval for excessive blast radius.

4.3 Data Quality Control: Dual-Layer QC Architecture

Centralized QC (Platform-level): Gatekeeper at ingestion. Dimensions: completeness, uniqueness, conformity, consistency. Checks: schema validation, format verification, deduplication, reference data reconciliation, non-null enforcement. Sidecar monitoring, non-blocking, alerts + remediation tickets.

Context-Aware QC (Dataset-level): Scenario-specific. Dimensions: accuracy, timeliness, domain consistency, annotation quality. Checks: feature drift detection, input-output consistency, domain constraint violations, annotation consistency. Embedded in pipeline, real-time anomaly detection.

Six-dimensional Data Quality Index (DQI) with weighted sum: completeness, accuracy, consistency, timeliness, uniqueness, compliance. Weights adjustable per scenario (medical imaging weights accuracy higher; real-time annotation weights timeliness).

Four-layer rule engine: Application (dashboards, reports, alerts, tickets), Execution (scheduling, batch run, aggregation, remediation tracking), Parsing (rule parsing, SQL generation, expression evaluation, AI inference), Storage (rule repo, metadata, execution history, problematic data). Four execution modes: real-time (strong validation), scheduled (daily patrols), on-demand (special governance), event-driven (pre-release/pre-training triggers).

4.4 Annotation Pipeline Deep Integration with Lakehouse

Traditional "annotation tool + file storage" creates silos: results stuck in tool DB, manual export/import, version chaos, consistency issues.

Unified raw data storage: Raw data in Paimon/Iceberg lake; annotation platform accesses via presigned URLs — no platform bottleneck, short TTL prevents permanent leakage.

Real-time annotation write-back: Streaming writes to Paimon tables, incremental queries; combined with AI pre-annotation for human-in-the-loop iterative flywheel.

Dataset versioning: Lake format snapshots for every change; lakeFS adds Git-like branching, merging, commits for annotation experiments and A/B comparisons.

Quality measurement loop: Post-ingestion QC engine auto-runs checks, writes annotator consistency, accuracy scores back to quality metadata tables — closing "annotate → inspect → score → improve" loop.

Challenge 3: Compute & Query Performance

5.1 Multi-Source Heterogeneous Fusion: Federated Query Breaks Silos

High-quality datasets originate from diverse sources: CDC-synced structured business data, object-stored unstructured files, annotation platform results, feature engineering vectors. Physically scattered, logically unified.

Federated Query via Catalog mechanism (StarRocks) manages internal tables and external lake tables simultaneously, supports cross-Catalog joins. Example: single SQL joins Paimon lake annotation metadata with StarRocks internal model evaluation results.

-- Register Paimon Catalog
CREATE EXTERNAL CATALOG paimon_catalog
PROPERTIES (
  "type" = "paimon",
  "paimon.catalog.type" = "filesystem",
  "paimon.catalog.warehouse" = "oss://my-datalake/paimon/",
  "paimon.s3.endpoint" = "oss-cn-beijing.aliyuncs.com",
  "paimon.s3.access_key" = "xxx",
  "paimon.s3.secret_key" = "xxx"
);
-- Cross-Catalog federated query
SELECT
  d.dataset_id,
  d.dataset_name,
  d.sample_count,
  m.model_name,
  m.accuracy,
  m.f1_score
FROM paimon_catalog.quality.datasets d
JOIN internal_db.model_evaluation m
  ON d.dataset_id = m.dataset_id
WHERE d.quality_score > 0.85
ORDER BY m.accuracy DESC
LIMIT 100;

5.2 Multi-Modal Retrieval Performance Optimization

Typical query: "Find all street-view images containing red sedans captured in daytime with annotation accuracy >0.9" — requires full-text search, vector search, scalar filtering, multi-route fusion and re-ranking. Traditional architecture calls three separate systems (search engine, vector DB, OLAP), causing redundancy, inconsistency, operational complexity.

Paimon + StarRocks unified solution via three mechanisms:

Global Row ID unified association: Paimon 2.0 assigns stable Global Row ID per row. Vector indexes, object files, scalar columns, full-text indexes all map to Row ID. Indexes decoupled from physical files; compaction changes layout but not logical identity.

Lake table global index model: Data File × Index Shard × Row ID Range. Multiple data files map to one index shard recording a Row ID range. Vector, full-text, B-Tree, Bitmap indexes built around Row ID ranges. Query returns Row IDs, then Manifest fetches columns.

Search & OLAP hybrid engine: StarRocks extends from pure OLAP to unified execution framework for full-text, vector, scalar filtering, hybrid recall, AI Functions. Scalar + vector dual-path recall with RRF/weighted fusion/re-ranking significantly improves dataset recall quality.

5.3 AI Training Data Access Optimization

Object storage has high capacity/low cost but poor small-file random access latency/throughput vs local storage, causing GPU starvation.

Tiered Storage Acceleration Architecture: Two-tier "cold data in lake + hot data in acceleration layer":

Data Lake Layer (Object Storage): Full persistence, low cost, high scale, Single Source of Truth.

Acceleration Layer (Parallel FS / Distributed Cache): Current training datasets cached on high-performance storage (all-flash arrays or local NVMe) for GPU-grade throughput/latency.

Automated Data Flow: Bucket Link connects object storage to acceleration layer; auto sync with incremental, selective, preload strategies; integrated with training scheduler.

Small-File Specific Optimizations:

Packing formats: TFRecord, WebDataset, Lance pack many small files into larger shards, reducing metadata ops. Lance delivers ~100x random point query speedup vs Parquet, 150x faster schema evolution.

Metadata caching: Acceleration layer caches file listings and metadata, avoiding massive LIST requests at training start.

Prefetch & pipelining: Overlap compute with next-batch prefetch; integrate with scheduler to load next job's data while current job trains, maximizing GPU utilization.

StarRocks Native Reader further boosts lake table reads: C++ native implementation eliminates JNI, Java type conversion, row-column transformation, GC overhead — >5x MOR table read throughput. Unified Data Cache manages page cache, metadata cache, lake file cache, vector index cache, narrowing lake vs internal table gap.

Challenge 4: Data Security & Compliance

6.1 Systematic Data Classification & Grading

High-quality datasets contain sensitive info: patient privacy (medical), biometrics (face), trade secrets (industrial). Regulations (Data Security Law, PIPL) mandate classification. Based on GB/T 38667-2020, a two-dimensional "impact object × impact severity" automated grading algorithm:

Three impact objects: national security, public interest, individual rights.

Three severity levels: general, serious, especially serious.

Formula: Security Level = f(max(impact object weight × severity coefficient)).

Four-tier security controls (illustrated in diagram). Automated identification & labeling: regex/keyword for structured sensitive fields (ID, phone, bank card, email); NLP semantic detection for unstructured text/annotations; linkage with data standards module — dictionary updates auto-sync to all referencing quality rules. In lakehouse, classification tags stored in unified metadata center, decoupled from underlying file formats — policies apply consistently across Paimon, Iceberg, etc.

6.2 Fine-Grained Access Control

Traditional warehouse permissions stop at table/column level. High-quality datasets need:

Row-Level Security (RLS): Annotators see only assigned tasks; hospital researchers see only their hospital's data.

Column-Level Security (CLS): Roles see different fields — analysts see feature vectors but not raw face images.

Dynamic Data Masking: No original data modification; query-time masking per role (e.g., dev environment hides middle 4 digits of phone).

Label-Based Access Control: Batch permissions via data labels (confidential/internal/public) — more efficient than per-table grants.

StarRocks implementation examples:

-- Row-level policy: annotator sees own tasks
CREATE ROW POLICY annotator_task_policy ON dataset_tasks
AS PERMISSIVE
USING (annotator_id = current_user());
-- Column-level: hide sensitive fields from analyst role
REVOKE ALL ON TABLE datasets.face_images FROM ROLE analyst;
GRANT SELECT(image_id, category, quality_score, create_time)
  ON TABLE datasets.face_images TO ROLE analyst;
-- Dynamic masking: mask middle 4 digits of phone
ALTER TABLE users
  MODIFY COLUMN phone SET MASKING POLICY mask_phone_inner;
CREATE MASKING POLICY mask_phone_inner AS (val VARCHAR(20))
  RETURNS VARCHAR(20) ->
  CASE WHEN current_role() = 'admin' THEN val
       ELSE CONCAT(LEFT(val, 3), '****', RIGHT(val, 4))
  END;

6.3 Full-Chain Audit & Traceability

Audit covers full lifecycle: data ingestion (who, when, how, what), queries (who, what SQL, tables/columns/rows, result volume), exports (who, destination, volume), changes (who, what data/metadata/policy, before/after), annotation operations (who, which samples, results, review status).

Implementation mechanisms: SQL audit via query engine logs; file operation audit via object storage access logs; API audit via data service logs (caller, params, result summary); annotation audit via platform operation logs (submit, modify, review). Best practice: ingest audit logs into lakehouse as "audit data lake" for compliance reporting, anomaly detection, risk trend analysis.

Reference Architecture: AI-Native Lakehouse for High-Quality Datasets

7.1 Data Ingestion Layer

Five ingestion modes: CDC real-time sync (Flink CDC, sub-second), batch import (Spark/Flink, full/incremental), streaming (Kafka → Paimon, exactly-once), multi-modal collection (images/video/audio via object storage SDK, metadata to lake tables), API integration (scheduled pull or event-driven). Core principle: high fidelity — ODS retains raw format, no business logic transformation, ensuring original payload available for audit. System-level cleansing limited to timestamp normalization, batch ID injection, source system tagging.

7.2 Lakehouse Storage Layer

Foundation: unified object storage + open table formats + four-tier hot/cold tiering.

Storage base: Object storage pool, compute-storage separation, elastic scaling.

Lake formats: Primary Apache Paimon (real-time updates + multi-modal), secondary Iceberg (cold archival + cross-engine compatibility), unified Catalog, auto-conversion by data temperature.

Multi-modal storage: Structured in Parquet/ORC, Blobs as independent objects, vectors in dedicated formats, all linked via Global Row ID.

Hot/cold tiering: Hot (SSD, <7d), Warm (OSS Standard, 7-90d), Cold (OSS IA, 90-365d), Archive (OSS Archive, >365d), auto-demotion and on-demand promotion.

Version management: Lake format snapshots for traceability; lakeFS for Git-like branching, merging, commits — supporting annotation experiments and dataset version iteration.

7.3 Data Governance Layer

Five subsystems ensuring "high quality":

Metadata Management: Unified technical/business/operational metadata, asset catalog & search.

Data Lineage: End-to-end (table → column → operator), impact analysis, root cause, change assessment.

Quality Control: Rule engine + AI-assisted dual-layer QC, six-dimensional DQI, four execution modes.

Security & Compliance: Classification, fine-grained permissions, dynamic masking, full-chain audit.

Annotation Pipeline: Lakehouse-integrated workflow: AI pre-annotation, human review, quality measurement, versioning.

Core design principle: native embedding — security policies and quality rules not external after-the-fact checks but embedded in development, query, export workflows: "develop = compliant, write = quality-checked."

7.4 Compute Engine Layer

Diverse workloads served by multiple engines sharing unified storage/metadata via Catalog, deployed on Kubernetes with elastic scaling and isolated resource pools:

Streaming (Flink): Real-time ingestion, annotation write-back, quality monitoring, feature computation.

Batch (Spark): Bulk processing, offline ETL, batch quality checks, training data preparation.

OLAP (StarRocks): Ad-hoc queries, BI analysis, multi-modal retrieval, dataset exploration.

Vector Retrieval: Integrated with StarRocks, IVF_PQ, HNSW indexes for semantic search & similarity matching.

7.5 Data Application Layer

Multi-faceted consumption: BI/visualization (reports, dashboards, large screens), self-service analysis (SQL, exploration, preview), AI model training (dataset export, feature extraction, training pipelines), data service APIs (standardized interfaces), data asset management (catalog, quality dashboards, lineage graphs, compliance reports).

Summary & Outlook

8.1 Core Conclusions

Explosive growth of high-quality datasets (120K datasets, 1,565 PB, 60% QoQ) challenges traditional lakehouse across four interconnected dimensions requiring systemic evolution, not point fixes. Response summarized as "Four Ones":

One Unified Foundation: Object storage + open table formats + compute-storage separation, tiered storage cuts cost by an order of magnitude.

One Governance System: Metadata-centric, integrating lineage, quality, security, annotation — native embedding, full-chain coverage.

One Compute Engine Suite: Streaming + batch + OLAP + vector retrieval, unified Catalog for data interchange and federated query.

One New Paradigm: From BI-oriented structured lakehouse to AI-native multi-modal lakehouse — Global Row ID + unified index model unify structured/unstructured/vector data.

8.2 Technology Trends

Lakehouse evolves to AI-native: Storage shifts from columnar-only to columnar+vector+object unified; compute adds vector search, full-text, AI Functions. Leaders pivot to "Data Intelligence Platform" with rising AI revenue share.

Lake-Stream unification boosts real-time: Paimon + Fluss + StarRocks fuse stream storage with lake storage — second-level freshness, 10x stream storage cost reduction, freshness doesn't accumulate across layers — critical for real-time annotation, online learning.

AI-driven automated governance: Shift from manual rule config to "AI-assisted + human-in-the-loop." AI auto-detects sensitive data, generates quality rules, finds anomalies, enriches metadata. Governance roles become AI trainers & result reviewers.

Data assetization & value release: As data property rights registration and factor markets mature, high-quality datasets transform from "resources" to "assets." Lakehouse adds asset valuation, rights registration, trading integration to support data element value release.

High-quality dataset construction is a long-term endeavor; lakehouse architecture will continuously evolve. Technology selection must adhere to open standards, avoid vendor lock-in; architecture design must enforce layered decoupling and incremental evolution; organizational governance must fuse data governance with business depth. Only then can data resources truly become data assets, providing a solid foundation for industrial intelligence in the AI era.