EMR Serverless StarRocks + DLF Paimon: Multimodal Data Lake Retrieval for Autonomous Driving

Multimodal Data Lake Retrieval: From Raw Objects to Intelligent Analysis

Enterprise data is shifting from structured tables to images, videos, text, and vectors. In autonomous driving, each test vehicle continuously generates road videos, sensor data, and operational logs, while algorithm teams produce labels, features, and evaluation results. Storing data is only the first step; the real challenge is enabling business users and engineers to find relevant content and analyze it.

The core difficulty: systems see a video as a file with path, size, and upload time, but the business cares about semantic content — pedestrians, rain, anomalous scenes. File metadata cannot answer these questions.

Overall Architecture: A Complete Pipeline from Raw Objects to Analysis

From Files to Queryable Data

Raw objects (videos, images) enter the platform and are processed: content parsing, scene description and tag generation, vector feature extraction. The original files remain in object storage, while processing results — text, tags, vectors — are linked to the original objects. This creates two layers: an immutable, large-volume raw object layer, and a queryable, searchable processed layer. Business users now face structured information that can be filtered by conditions, matched by keywords, and semantically matched.

Storage Selection by Access Pattern

Frequently accessed, low-latency data goes into StarRocks internal tables. Long-term, large-scale data assets are managed via DLF Paimon lake tables, leveraging snapshots and continuous changes. Existing Lance-format data can be directly accessed via EMR Serverless StarRocks for retrieval and analysis without migration.

Retrieval Is Not the Endpoint

After finding content, businesses need further analysis: how many such scenarios, which vehicle models, which model versions. Retrieval results must join with business data for mixed analysis, so retrieval and downstream analysis are designed within the same query process.

Two Core Design Principles: Early Filtering, On-Demand Reading

Early Filtering: Exclude Irrelevant Data as Soon as Possible

A query may contain structured conditions (time, vehicle model) and unsupervised conditions (keywords, vector similarity). The system decides which conditions to execute first, where to fetch candidate sets, and when to read full data. Example: to find a vehicle model's rainy-night driving clips from the last month, the structured conditions should be applied early to drastically reduce the candidate set before vector computation or full-content reads.

On-Demand Reading: Fetch Identifiers and Scores First, Large Fields Later

During retrieval, the system first passes candidate identifiers and relevance scores. Only after the candidate set shrinks does it read the business fields and original object references needed for display and analysis. The larger the images/videos, the more valuable this approach, because reading large fields costs far more than computing conditions and scores.

Together, these principles unify different retrieval modes into a single query execution process while considering subsequent joins and analysis, minimizing data movement across systems.

A Single SQL Illustrates the Full Picture

SELECT
    clip_id,
    oss_uri,
    model_version
FROM DLF Paimon.driving_clips
WHERE tenant_id = 1001
  AND scene_text MATCH 'rainy night cut-in'
ORDER BY hybrid_score(
    embedding,
    :query_vector
)
LIMIT 20;

Execution: the system determines the data range and snapshot, performs hybrid analysis of structured conditions with full-text and vector search to form candidates, then reads additional business fields. Crucially, the system sees the complete query, enabling global trade-offs at each stage: if the business scope is small, filter early; if further ranking is needed, retain a suitable candidate size; if only a few results are returned, defer reading large fields. Moreover, the query doesn't end at video retrieval — it can continue joining model versions, evaluation metrics, and business records, turning a multi-system workflow into a single query chain.

Managing and Querying Lake Data with DLF Paimon

Lake Query Requirements

Users want to query lake data with familiar SQL: time filters, joins with model info tables, aggregations per version. In this solution, Paimon's Global Index manages lake table data, snapshots, and continuous changes, while StarRocks handles query and analysis. A lake table can associate stable business fields, dynamic model-generated attributes, text, vectors, and object references. The underlying lake-table access paths are handled by EMR Serverless StarRocks, not exposed to users.

Global Index: Like a Book's Table of Contents

To query a thick book, you check the index first, locate relevant pages, then read only those pages. Global Index works the same way: Point lookup by sample ID → direct location via index (like page number). Text search or semantic similarity → get candidate identifiers and relevance scores (like marking potentially relevant pages). Merge candidates, then read required business columns (like reading only the marked pages).

For massive data, this is highly valuable: users may need only dozens of results, so the read scope should be minimized to relevant data. Implementation concepts like Row ID, Index, Split are internal; the core idea is locate candidate set first, then read on demand.

Multiple Facets of a Sample Enable Natural Combined Queries

A single driving video has many facets: fixed fields (collection time, vehicle model), model-generated scene descriptions, continuously added tags and vector features. Once linked, retrieval methods can be chosen per question:

Exact ID → point query.

Specific keywords → full-text search.

Natural language description or reference sample → vector semantic search.

Real workloads often combine these: restrict by vehicle model and time, then find clips matching a scene description. Keeping them in one data relationship makes combined queries natural — no export/import across systems.

Choosing Between DLF Paimon and Lance

If you already have a Flink + DLF Paimon lakehouse with continuous updates and business analytics, prefer DLF Paimon. Adding multimodal retrieval enhances the existing stack rather than starting from scratch.

If your workflow revolves around Python, Arrow, or training datasets already in Lance format, and you prioritize row-wise reads and feature engineering, evaluate Lance. Its design aligns with such workflows.

Final choice depends on actual read/write patterns, connector support, and operational cost — no one-size-fits-all answer.

An EMR Serverless StarRocks retrieval platform can unify the query entry while allowing different business units to choose different lake formats underneath.

Production Practices: From Alibaba Group to Cloud Industries

Alibaba Group A Lake: Pre-Training Data Retrieval and Sample Preparation

EMR Serverless StarRocks serves pre-training data retrieval and sample preparation. Training datasets contain text, images, audio, video, with continuously generated labels and features. A Lake uses DLF Paimon wide tables to associate multimodal samples with features, and StarRocks provides hybrid retrieval and filtering. Example: prepare a specialized training set by constraining source, quality, scene, then keyword or semantic search for candidates. Retrieval participates in training data selection, helping engineers quickly find data worth further evaluation.

AI Data Platform: Two Distinct Retrieval Needs

Large-scale candidate recall: 10B+ vectors, top 5M recall for audience targeting. Unlike typical search returning tens of results, candidate volume is huge, and recall is followed by further filtering, joining, and analysis, demanding high query-execution throughput.

Agent observability: Troubleshoot an agent invocation by request/trace ID, narrow by time, tool, status, then find similar issues via error text or semantic similarity.

Both scenarios show that hybrid retrieval combining vector, full-text, and scalar conditions is now common; single retrieval modes rarely cover real needs alone.

Cloud Industries: Different Data, Same Challenge

Automotive (cockpit multimodal point queries and filtering), smart hardware (organizing training data around raw objects), media & entertainment (linking shot parsing, feature generation, and material recall). Data differs, but the shared challenge is linking raw content, model-generated information, and business records — exactly where this solution adds value.

Deep Dive: Autonomous Driving Closed Loop

Autonomous driving best illustrates multimodal data value: vehicles continuously produce video, sensor data, logs; algorithm teams continuously generate labels, features, evaluations. As data grows, engineers face: when a model exhibits a certain problem, how to find truly relevant samples from existing data?

Step 1: Locate and Analyze Bad Cases

A bad case is a sample where behavior deviates from expectations. Engineers start from a takeover event or evaluation failure record, locate the video clip, and associate the model version and processing logs. The description "rainy night cut-in" is broad; concrete conditions emerge: heavy rain, strong road reflection, a vehicle cutting in at short distance with partial occlusion. Clarifying these conditions makes subsequent data search targeted. Structured analysis and video content must work together: see what happened, know when, which vehicle model, which model version. Further statistics: is the issue concentrated in one version or across multiple versions?

Step 2: Find Similar Historical Samples

Relying solely on manual labels is problematic: same situation labeled "cut-in" vs "vehicle lane change"; early data lacks fine-grained scene tags. A single keyword search misses valuable data. Combine retrieval methods: first restrict by vehicle model and collection time, then full-text search on existing scene descriptions and tags, finally vector search for content-similar clips. Engineers can input "rainy night close-distance vehicle cut-in" or use a discovered problematic clip as reference. The choice depends on the model and data pipeline. Returned results are further analyzed for distribution: concentrated on a few routes? A few vehicles? Coverage across rain intensities and lighting? This step uses joins and aggregations to ensure the training candidate set covers needed scenarios, not just near-duplicate videos.

Step 3: Reuse or Targeted Collection

If suitable samples exist, enter manual relabeling and quality check. Raw clips stored earlier can be rediscovered with new tags, descriptions, features for new specialized tasks — same raw data continuously yields new uses as model capabilities and business needs evolve. If coverage is insufficient, turn missing conditions into targeted collection requirements: specific weather, lighting, road environments, with standardized collection and quality inspection. Note: a retrieval miss may stem from label, feature, or query condition issues; verify retrieval scope and data processing before concluding data is truly missing. This avoids redundant collection of already-rich scenarios.

Step 4: Training, Evaluation, and Version Management

After sample completion, training and evaluation require precise traceability: which data was used? Which label version? Which samples are new? Associate raw clips, processing records, annotation results around a business data ID, and persist dataset versions. Distinguish two levels: DLF Paimon snapshots manage underlying data state; business must still record training set composition and its mapping to model versions. When a new model's performance changes, engineers can analyze whether it's due to data changes, training config, or other factors. Evaluation goes beyond overall metrics to the specific scenario: did the new model improve on rainy night cut-in? Any regression in other scenarios? Scenario libraries and bad case libraries provide continuous regression checks for future versions.

After Closure: Data Grows Richer with Use

Post-evaluation, the model enters canary release and feedback loops. New problem clips feed the next analysis cycle. Over time, the platform accumulates raw samples, scene descriptions, multiple training set versions, evaluation results. Linking these assets enables efficiency analysis: time from problem discovery to reusable training set; time spent on search, labeling, quality check; reuse ratio of existing data; which scenarios always need supplemental collection. These questions are answerable via data, helping teams improve R&D efficiency.

Summary: Helping Engineers Answer Four Questions

Multimodal data in enterprises needs a complete usage chain: extract information from raw objects, organize samples with business records, retrieve relevant content via search, and continue analysis. EMR Serverless StarRocks and DLF Paimon collaborate around this chain. In autonomous driving, the division is clear: DLF Paimon manages data assets and snapshots; StarRocks enables cross-modal sample retrieval and joins with model, evaluation, and process records for analysis. Together with collection, labeling, training, and deployment platforms, they close the loop. Data value manifests in every problem localization, sample supplementation, and model evaluation.

Ultimately, this data architecture helps engineers answer four concrete questions faster:

Where did the problem occur?

Where is the relevant data?

Is the data sufficient?

Has the improvement been validated?

This is the most direct value of multimodal retrieval in autonomous driving — enabling teams to leverage existing data assets more conveniently, giving model iterations a stronger data foundation.