Skip to content
StarRocks + Paimon: Multimodal Data Lake Retrieval Architecture
Databases 26 min read

StarRocks + Paimon: Multimodal Data Lake Retrieval Architecture

This article details how StarRocks and Paimon combine to build a multimodal data lake retrieval system, covering architecture design, early filtering and on-demand reading principles, Paimon's Global Index, Paimon vs Lance selection, and real-world implementations in autonomous driving and AI data platforms at Alibaba.

StarRocks
StarRocks
StarRocks

01 The Challenge: Data Stored, Now What?

Enterprise data is shifting from structured tables to multimodal assets—images, videos, text, vectors. In autonomous driving, each test vehicle generates continuous road video, sensor data, and logs, while algorithm teams produce labels, features, and evaluation results. Storing data is only the first step; the real problem is enabling business users and engineers to find relevant content and analyze it.

The difficulty stems from a mismatch: systems see a video as a file with path, size, and upload time, while the business cares about semantic content—pedestrians, rain, anomalous scenes. File metadata cannot answer these questions.

02 Overall Architecture: From Raw Objects to Analyzable Data

The solution connects raw object processing, storage, retrieval, and analysis into a single pipeline:

1. From Files to Queryable Data

Raw videos and images enter the platform, undergo parsing, scene description, tag generation, and vector extraction. The original objects remain in object storage, while processed results—text, tags, vectors—are linked to them. This creates two layers: an immutable raw-object layer and a queryable processed layer.

Multimodal data lake retrieval architecture: unified query entry, diverse storage, StarRocks orchestrates execution
Multimodal data lake retrieval architecture: unified query entry, diverse storage, StarRocks orchestrates execution

2. Choose Storage by Access Pattern

Frequently accessed, low-latency data goes into StarRocks internal tables. Long-term, large-scale assets are managed via Paimon lake tables, leveraging snapshots and continuous changes. Existing Lance-format data can be queried directly through StarRocks without migration.

3. Retrieval Is Not the Endpoint

After finding content, businesses need further analysis—counts, distribution by vehicle model, correlation with model versions. Retrieval and downstream analysis are designed as a single query process, avoiding round-trips between systems.

03 Two Core Design Principles: Early Filtering, On-Demand Reading

Architecture diagram: unified query entry, diverse storage, StarRocks orchestrates execution
Architecture diagram: unified query entry, diverse storage, StarRocks orchestrates execution

1. Early Filtering: Exclude Early

A query may combine structured conditions (time, vehicle model), keywords, and vector similarity. The system decides which conditions to apply first, where to fetch candidate sets, and when to read full data. Example: finding a vehicle model's rainy-night clips from the last month—structured filters should shrink the candidate set before expensive vector computation.

2. On-Demand Reading: IDs and Scores First, Large Fields Later

During retrieval, the system passes candidate identifiers and relevance scores. Only after the candidate set is sufficiently narrowed does it fetch display fields and original object references. This is critical for large images/videos where read cost dominates.

Unified SQL Execution View

A single SQL expresses the full need: find driving clips related to "rainy night cut-in" within a business scope, return top matches.

SELECT clip_id, oss_uri, model_version
FROM paimon.driving_clips
WHERE tenant_id = 1001
  AND scene_text MATCH '雨夜 加塞'
ORDER BY hybrid_score(embedding, :query_vector)
LIMIT 20;

Execution: determine data range and snapshot, mix structured filters with full-text/vector search to form candidates, then read required business fields. The optimizer sees the whole query, enabling global trade-offs—early convergence for small scopes, preserving candidate size for ranking, deferring large-field reads for small result sets. Results can further join model versions, evaluation metrics, and business logs, turning a multi-system workflow into one query chain.

04 Managing and Querying Lake Data with Paimon

Lake Query Expectations

Users want familiar SQL—time filters, joins with model tables, aggregations—on lake data. Paimon's Global Index manages data, snapshots, and continuous changes; StarRocks handles query and analysis. A lake table can hold stable business fields, dynamic model-generated attributes, text, vectors, and object references, all linked.

Global Index: Like a Book's Table of Contents

Point lookup by sample ID → direct location via index (like page number).

Text or semantic search → get candidate IDs and relevance scores (like marking relevant pages).

Merge candidates, then read only needed business columns (like reading only marked pages).

For massive data, this avoids full scans; users may need only dozens of results.

Combined Queries Across a Sample's Facets

A single driving video has fixed fields (time, vehicle), scene descriptions, evolving tags, and vector features. The system supports point lookup, full-text search, and vector similarity—often combined, e.g., filter by vehicle and time, then search by scene description. Keeping these in one relational structure makes hybrid queries natural without export/import across systems.

05 Selection Guide: Paimon or Lance?

Choose Paimon if you already have a Flink + Paimon lakehouse with continuous updates and need business analytics. Adding multimodal retrieval enhances the existing stack.

Evaluate Lance if your workflow centers on Python, Arrow, or training datasets already in Lance, especially for row-oriented reads and feature engineering.

Final decision should weigh actual read/write patterns, connector support, and operational cost. A unified StarRocks retrieval platform can query multiple lake formats underneath.

06 Production Deployments: From Alibaba Group to Cloud Industries

1. Alibaba Group A Lake: Pre-Training Data Retrieval & Sample Preparation

Training datasets contain text, images, audio, video, with continuously generated labels and features. A Lake uses Paimon wide tables to link multimodal samples with features, and StarRocks for hybrid retrieval and filtering. Example: prepare a specialized training set by constraining source, quality, scene, then keyword/semantic search for candidates. Retrieval participates in data selection—checking relevance, coverage, deduplication, and identifying samples covering model weak spots.

2. AI Data Platform: Two Distinct Retrieval Workloads

Large-scale candidate recall: 10B+ vectors, top 5M recall for audience targeting. Differs from typical top-10 search; huge candidate sets require further filtering, joining, analysis, demanding high-throughput query execution.

Agent observability: Locate by request/trace ID, narrow by time/tool/status, then find similar issues via error text or semantic similarity.

Both illustrate that hybrid search (vector + full-text + scalar) is now the norm; single-mode retrieval rarely suffices.

3. Cloud Industries: Different Data, Same Challenge

Automotive (cockpit multimodal point lookup/filtering), smart hardware (training data around raw objects), media & entertainment (shot parsing, feature generation, material recall). All face the same core challenge: linking raw content, model-generated info, and business records.

07 Deep Dive: Autonomous Driving Closed Loop

Closed loop from bad case localization to sample augmentation, training, and evaluation
Closed loop from bad case localization to sample augmentation, training, and evaluation

Scenario: a model version underperforms on rainy-night cut-in during road test. Need to understand conditions and find supporting historical data.

Step 1: Locate and Analyze Bad Cases

Engineers start from a takeover event or failed evaluation record, retrieve the video clip, and correlate model version and processing logs. The vague "rainy night cut-in" is refined: heavy rain, strong road reflection, adjacent vehicle cuts in at short distance with occlusion. Structured analysis and video review combine to quantify issue distribution across versions.

Step 2: Find Similar Historical Samples

Manual tags are inconsistent ("cut-in" vs "vehicle merging") and early data lacks fine-grained labels. Combine retrieval modes: filter by vehicle model and collection time, full-text search on scene descriptions, vector similarity on content. Engineers can input "rainy night close-distance vehicle cut-in" or use the problematic clip as reference. Results are further analyzed via joins and aggregations—distribution by route, vehicle, rain intensity, lighting—to ensure candidate set covers target scenarios, not just near-duplicates.

Step 3: Reuse or Targeted Collection

Reuse: Existing suitable samples enter manual relabeling and quality check. Raw clips stored earlier gain new utility through fresh tags, descriptions, features for new tasks.

Targeted collection: If gaps remain, define precise collection requirements (weather, lighting, road types) with standardized acquisition and QC.

Crucially, a retrieval miss may stem from labeling, feature, or query issues—teams must audit retrieval scope and processing before concluding data is truly missing.

Step 4: Training, Evaluation, Version Management

After sample augmentation, training and evaluation require exact traceability: which data, which label version, which samples are new. Business-level data IDs link raw clips, processing records, annotation results, and dataset versions. Paimon snapshots manage underlying data state; business layer records training set composition and model-version mapping. This enables root-cause analysis when model behavior changes—data shift vs. config shift. Evaluation goes beyond overall metrics to the specific scenario: did rainy-night cut-in improve? Any regression on existing scenarios? Scenario and bad-case libraries provide continuous regression checks.

Step 5: Post-Closure—Data Grows Richer

After deployment, new issues feed the next cycle. Accumulated assets include raw samples, scene descriptions, dataset versions, evaluation results. Linking them enables efficiency analysis: time from issue to reusable dataset, bottlenecks (search, labeling, QC), reuse rate, scenarios always needing new collection—all measurable for R&D improvement.

08 Summary: Answering Four Questions

Multimodal data needs a complete usage chain: extract information from raw objects, organize samples with business records, retrieve across modalities, and analyze. StarRocks and Paimon divide labor: Paimon manages data assets and snapshots; StarRocks enables cross-modal sample retrieval and joins with model, evaluation, and process records for analysis. Together with collection, labeling, training, and deployment platforms, they close the loop.

The architecture aims to help engineers answer four concrete questions faster:

Where did the problem occur?

Where is the relevant data?

Is the data sufficient?

Has the improvement been verified?

This is the direct value of multimodal retrieval in autonomous driving—making existing data assets more accessible and giving model iterations stronger data backing.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

StarRocksvector searchPaimonautonomous drivinglakehouseGlobal Indexhybrid searchmultimodal data lake
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.