Unstructured Data Lake Ingestion: File Body + Metadata Registration Patterns

Core Consensus: File Body + Metadata Must Both Enter the Lake

The industry consensus is that storing only files makes governance impossible, while storing only metadata makes data unusable. Combining both enables unstructured data to be manageable, usable, and traceable — the foundation for AI-ready data lakes.

Industry-Standard Three-Layer Architecture

A typical architecture adopts three layers to meet large-scale storage and governance needs for AI scenarios:

Storage Layer — Handles file body storage (S3/HDFS/MinIO), providing high-throughput, low-cost object storage.

Metadata Layer — Manages structured metadata, usually built on Paimon/Hive as a unified metadata lake, recording file paths, tags, business attributes, etc.

Ingestion Layer — Synchronously writes files and metadata, providing batch/real-time ingestion capabilities with data quality validation tools.

Batch Ingestion (Historical AI Data, Offline Collection)

Step 1: Define File Storage Path Convention (Critical Prerequisite)

The industry-standard path convention follows a three-level structure: business_line/data_type/time_partition, ensuring manageable files. Example:

# Format: /warehouse/{business_line}/{data_type}/{time_partition}/{filename}.{suffix}
# Example: AI training image data
/s3/paimon_warehouse/ai_train/image/2025-12-12/cat_001.jpg
# Example: User behavior log text
/hdfs/paimon_warehouse/user_behavior/text/2025-12/click_log_20251212.txt

Key points: time partition granularity by day/week to avoid excessive files per directory; filenames include business identifier + timestamp to prevent duplicates (e.g., model_v3_train_data_20251212_001.csv).

Step 2: Batch Upload Files to Storage Layer

Tool selection : small files use rclone / aws s3 cp; large files use distcp (HDFS-to-HDFS) / multipart upload (S3).

Critical operation : after upload, generate a unique file path (e.g., s3://paimon_warehouse/ai_train/image/2025-12-12/cat_001.jpg) and compute file MD5 for integrity verification.

Step 3: Structured Metadata Registration (Core Step)

Create metadata management table (using Paimon for ACID and incremental updates):

CREATE TABLE ai_unstructured_meta (
  file_path STRING COMMENT '文件存储路径，主键',
  file_name STRING COMMENT '文件名',
  file_type STRING COMMENT '数据类型：image/text/audio/video',
  business_line STRING COMMENT '所属业务线',
  data_label STRING COMMENT 'AI标签，多个用逗号分隔',
  file_size BIGINT COMMENT '文件大小（字节）',
  md5 STRING COMMENT '文件校验值',
  create_time TIMESTAMP COMMENT '入湖时间'
) WITH (
  'connector' = 'paimon',
  'warehouse' = 's3://paimon_warehouse',
  'database' = 'meta_db',
  'table' = 'ai_unstructured_meta',
  'primary-key' = 'file_path'
);

Batch write metadata :

-- Flink SQL batch import
INSERT INTO ai_unstructured_meta
SELECT file_path, file_name, file_type, business_line, data_label, file_size, md5, create_time
FROM csv_table -- read metadata CSV file
WHERE file_path IS NOT NULL;

Generate metadata CSV manifest (containing file paths, tags, etc.).

Import into metadata table via Flink/Spark batch jobs.

Step 4: Data Integrity Verification

Verification logic: compare uploaded file count vs. metadata table record count and file MD5 vs. metadata MD5 .

Tool: write Python scripts to automate verification (traverse storage paths, query metadata table for comparison).

Real-Time Ingestion (Real-Time AI Data: Camera Streams, Real-Time User Behavior Text)

Core is real-time file write + real-time metadata registration , relying on stream processing engines.

Step 1: Deploy Real-Time Collection Tools

Structured logs : use Flink CDC to collect business system logs, output as text files.

Unstructured stream data : use Flume/Kafka Connect to collect image/audio streams, slice by time to generate files (e.g., one audio file every 5 minutes).

Step 2: Stream Write to Storage Layer + Metadata Layer

Using Flink as an example, implement dual-stream synchronization of file write and metadata registration :

// 1. Read real-time unstructured data stream (e.g., image binary data in Kafka)
DataStream<UnstructuredData> dataStream = env.addSource(new KafkaSource<>("ai_image_topic"));
// 2. Split processing: file write + metadata generation
dataStream.process(new ProcessFunction<UnstructuredData, Tuple2<String, MetaData>>() {
  @Override
  public void processElement(UnstructuredData data, Context ctx, Collector<Tuple2<String, MetaData>> out) {
    // a. Write binary data to S3, generate unique path
    String filePath = "s3://paimon_warehouse/ai_train/image/" + data.getDate() + "/" + data.getFileName();
    s3Client.putObject(filePath, data.getBinaryData());
    // b. Generate metadata object
    MetaData meta = new MetaData();
    meta.setFilePath(filePath);
    meta.setFileType("image");
    meta.setDataLabel(data.getLabel());
    meta.setCreateTime(new Timestamp(System.currentTimeMillis()));
    // c. Output file path and metadata
    out.collect(Tuple2.of(filePath, meta));
  }
});
// 3. Write metadata to Paimon metadata table
metaDataStream.addSink(new PaimonSink<>("meta_db.ai_unstructured_meta"));

Step 3: Real-Time Monitoring & Alerting

Monitoring metrics: file write success rate, metadata registration latency, file size anomalies.

Alert triggers: when metadata registration latency > 1 minute, or file MD5 verification fails.

AI-Scenario Special Optimizations (Industry Best Practices)

Label System Standardization

Establish a unified dictionary for AI data labels (e.g., three-level label cat:animal:pet) to avoid label chaos.

The data_label field in the metadata table stores multiple labels in JSON format (e.g., {"category":"cat", "scene":"indoor"}).

File Format Optimization

Image data: convert to WebP format to reduce storage cost.

Text data: convert to Parquet format to improve AI model read efficiency.

Large files: use sharded storage (e.g., video files >10GB split into multiple 1GB shards).

Integration with AI Models

Add model_version field in metadata table to link the AI model version trained on the data.

Enable intelligent retrieval based on metadata: e.g., "query cat images used to train model v3 in December 2025" — filter directly via metadata table without scanning all files.

Key Implementation Considerations

Permission Control : Storage layer (S3/HDFS) and metadata layer permissions must be consistent to avoid "can query metadata but cannot access files".

Version Management : Paimon metadata tables support versioning, enabling traceability of file history versions (e.g., AI data iterative updates).

Cost Control : Migrate low-frequency-access AI historical data to low-cost storage (e.g., S3 Infrequent Access), and mark storage tier in the metadata table.