Stella + Paimon: Making Multimodal Data Lakes Searchable and Analyzable with SQL

The article opens with a concrete autonomous-driving scenario: a rainy-night cut-in video sits in OSS, yet the model still fails on similar cases. The core problem is not storage but the ability to find similar historical clips, know their vehicle models and model versions, and feed them back into training and evaluation.

Architecture: Three Coordinated Layers

For Alibaba Cloud users the pipeline splits into three layers:

OSS raw objects – keep original videos, key frames, sensor segments; no need to copy large files into analytical tables.

DLF Paimon lake tables – manage sample metadata, descriptions, tags, embeddings, snapshots, and indexes. Columns include clip_id, vehicle model, timestamp, scene text, embedding, model version.

EMR Serverless StarRocks (Stella kernel) – queries lake tables via DLF Catalog, combines scalar filters, full-text and vector search, and joins with business tables for aggregation and analysis.

The key value is placing "find samples" and "explain samples" in a single SQL chain. After retrieving top-K vectors, engineers can further aggregate by route, vehicle model, software version, and issue type without moving candidate IDs to another system.

Stella vs. StarRocks

StarRocks is a MySQL-protocol analytical database for real-time analytics, lakehouse queries, and multi-dimensional OLAP. EMR Serverless StarRocks is the fully managed cloud service. Stella is the enterprise-grade kernel inside that service, not a separate database. Alibaba Cloud versions explicitly label the Stella kernel version to identify enterprise optimizations.

In storage-compute separation instances, compute nodes handle queries and ingestion while data resides in shared storage. Compute groups let interactive retrieval, batch data preparation, and other analytical tasks run in separate resource pools while sharing the same underlying data – useful for multimodal workloads where material processing has batch peaks but online retrieval needs stable latency.

Stella continuously enhances lake-table reads, caching, query optimization, and multimodal capabilities. For this article the critical features are: direct access to Paimon lake tables via DLF Catalog, vector search on available Global Indexes with push-down of structured predicates, and subsequent joins/aggregations/sorts on results. Internal-table vector indexes are also supported; existing internal-table data can use the same SQL entry point.

Version boundaries are explicit: DLF Paimon lake-table vector search requires EMR Serverless StarRocks 3.5.16-2.2.0 or later with a usable Paimon-side vector index; OSS Object Table object-manifest access starts at 3.5.16-2.2.2. Production rollout must validate against the actual running instance version and enabled capabilities.

From Files to Queryable Business Assets

After a video lands in OSS, the platform assigns a traceable clip_id linking the original URI, collection time, vehicle model, route, event, model version, and evaluation records. Content-processing jobs generate key frames, scene descriptions, tags, and embeddings, recording both the model and processing version so that re-labeling or re-embedding with a new model remains explainable.

On Alibaba Cloud the lifecycle is arranged as:

Raw content stays in OSS. Large files are not copied wholesale into analytical tables; tables store object references plus metadata needed for retrieval.

Processing results go into Paimon. DLF manages lake tables uniformly; Paimon snapshots give clear version boundaries for data processing, index building, and downstream consumption.

Stella handles query and analysis. A single SQL can restrict tenant, time window, vehicle model, hit text conditions, then rank by vector similarity, and finally join with evaluation tables.

AI Functions can participate in content processing. For instances with the capability enabled, SQL can invoke models to describe, classify, or vectorize objects; model outputs must still record processing version and quality state – a one-time generation is not treated as immutable truth.

For data already in Paimon, Stella queries directly via DLF Catalog without importing the whole table into an internal table. For frequently accessed, latency-sensitive subsets, internal tables and indexes can be evaluated. Both paths can coexist based on data temperature, write frequency, and query SLA.

Hybrid Retrieval Execution Walk-Through

Returning to the rainy-night cut-in: the query is not just "similar to this video". Business may also restrict vehicle model, time window, tenant, and require text matches for "rainy night" or "cut-in". Computing distances over all vectors first then filtering wastes work on irrelevant data; taking a tiny vector top-K first may leave too few candidates after filtering.

The preferred execution path: first pin the lake-table snapshot, then let the optimizer identify pushable scalar predicates and the top-K vector expression, narrowing candidates on the Paimon Global Index. The index returns row IDs and scores; business columns and OSS object references are fetched only as needed. For combined full-text, vector, and scalar conditions, actual push-down and index hits must be verified with EXPLAIN and Query Profile – not inferred from SQL surface form.

This "locate candidates first, then read large fields" order suits video and image workloads. The final result may be only dozens of clips; URIs, descriptions, and evaluation metrics should be read after the candidate set converges. The query's value is not just speed: it keeps candidate results and business dimensions inside SQL for further correlation, answering "which vehicle models or versions concentrate similar issues?"

Three SQL Patterns: Find Clips, Then Explain Them

Assume DLF Catalog connects to Paimon table dlf_catalog.driving.clips with columns clip_id, tenant_id, event_time, car_model, scene_text, embedding, oss_uri, model_version. {{QUERY_VECTOR}} is a client-side placeholder for a vector matching the index dimension and distance metric; MATCH, vector functions, and index push-down must be validated against the target version and Paimon index configuration.

1. Retrieve rainy-night cut-in clips within a business scope

SELECT clip_id, oss_uri, model_version, event_time,</code><code>       approx_cosine_similarity(embedding, {{QUERY_VECTOR}}) AS similarity</code><code>FROM dlf_catalog.driving.clips</code><code>WHERE tenant_id = 1001</code><code>  AND event_time >= '2026-10-01 00:00:00'</code><code>  AND event_time <  '2026-11-01 00:00:00'</code><code>  AND car_model = 'model_a'</code><code>  AND scene_text MATCH '雨夜 加塞'</code><code>ORDER BY similarity DESC</code><code>LIMIT 100;

LIMIT 100

is a target, not a guarantee of 100 qualified clips. Acceptance should record actual rows returned, post-filter cardinality, Recall@100, P50/P95 latency, index hit status, and snapshot version. Retrieval quality must be verified against human-labeled or trusted evaluation sets, not similarity scores alone.

2. Check coverage of the candidate set

WITH candidates AS (</code><code>  SELECT clip_id, route_id, car_model, model_version,</code><code>         approx_cosine_similarity(embedding, {{QUERY_VECTOR}}) AS similarity</code><code>  FROM dlf_catalog.driving.clips</code><code>  WHERE tenant_id = 1001</code><code>    AND scene_text MATCH '雨夜 加塞'</code><code>    AND event_time >= '2026-10-01 00:00:00'</code><code>    AND event_time <  '2026-11-01 00:00:00'</code><code>  ORDER BY similarity DESC</code><code>  LIMIT 500</code><code>)</code><code>SELECT car_model, model_version,</code><code>       COUNT(*) AS clip_count,</code><code>       COUNT(DISTINCT route_id) AS route_count,</code><code>       AVG(similarity) AS avg_similarity</code><code>FROM candidates</code><code>GROUP BY car_model, model_version</code><code>ORDER BY clip_count DESC;

This answers whether results concentrate in a few vehicle models or routes, avoiding the mistake of treating many near-duplicate clips as sufficient coverage.

3. Join candidates with model evaluation records

SELECT c.model_version,</code><code>       e.error_type,</code><code>       COUNT(*) AS affected_clips,</code><code>       AVG(e.score) AS avg_score</code><code>FROM dlf_catalog.driving.clips AS c</code><code>JOIN analytics.eval_results AS e</code><code>  ON c.clip_id = e.clip_id</code><code> AND c.model_version = e.model_version</code><code>WHERE c.tenant_id = 1001</code><code>  AND c.scene_text MATCH '雨夜 加塞'</code><code>  AND c.event_time >= '2026-10-01 00:00:00'</code><code>  AND c.event_time <  '2026-11-01 00:00:00'</code><code>GROUP BY c.model_version, e.error_type</code><code>ORDER BY affected_clips DESC;

These three SQLs answer "where are the samples", "is coverage adequate", and "which model versions share the problem". Production tables should join on clip_id, model version, and processing version together to avoid double-counting from re-evaluation or re-embedding. Tenant permissions and object access control must also be enforced so retrieval results never cross business boundaries.

Choosing Between Paimon, StarRocks Internal Tables, and Lance

Multimodal data need not use a single storage format. Selection starts with existing pipelines and primary access patterns, then decides whether additional replicas are justified:

Existing Flink/DLF/Paimon real-time lakehouse with continuous updates – Keep Paimon lake tables, queried directly by Stella. Key acceptance points: snapshot and index consistency, predicate push-down, read/write parallelism, query latency.

Frequently accessed, strict online SLA hot subsets – StarRocks storage-compute separation internal tables. Key acceptance points: index build/maintenance, hot cache, concurrent P95/P99, write freshness.

Python/Arrow/training-sample pipelines already on Lance – Evaluate Stella access to Lance. Key acceptance points: connector capability, index and filter support, fetch columns, version compatibility.

Paimon suits organizing continuously changing samples, labels, and business records into shareable lake tables; StarRocks internal tables suit concentrated query service on hot data; existing Lance data should first be assessed for direct reuse. Final decision should compare cold/hot query latency, write latency, recall quality, storage replicas, and maintenance cost using the same business SQL – not extrapolate from a single fastest runtime.

From Group Practice to Cloud Industry: Same Method, Different Data

Alibaba Group's ALake platform, shared publicly, centers on pre-training data retrieval and sample preparation. It links text, image, audio, and video samples and features in Paimon wide tables, then uses StarRocks for hybrid retrieval and filtering. Algorithm engineers not only find similar content but also check sample provenance, duplication, and scene coverage to decide which clips enter specialized training sets.

Another internal practice from the AI Data platform: some tasks need one-shot recall of large candidate sets followed by further filtering and joining; others need to locate agent invocations by request ID or trace ID, then trace similar issues through time, tool, status, and error text. These show multimodal retrieval covers both similar-content search and large-candidate analysis plus observability queries. Cloud automotive, smart hardware, and media teams face different data but can reuse the "object – processing result – business record" association method.

Group practices provide design experience. Alibaba Cloud projects must validate the same methods against actual instance versions, available indexes, and resource specs before deciding whether data goes into Paimon lake tables, StarRocks internal tables, or remains in existing Lance pipelines.

Bad Case to Next Model Version: Landing the Closed Loop

Step 1: Locate the issue. Start from a takeover event or evaluation failure, fix the original video URI, time, vehicle model, model version, and processing version. Human confirmation of actual rainy-night cut-in conditions – rainfall, road reflection, occlusion, cut-in distance – makes subsequent retrieval verifiable.

Step 2: Find historical samples. Use structured conditions to scope, full-text search to cover varied descriptions, then vector similarity to surface linguistically unexpressed near matches. Aggregate returned results by route, vehicle, and time; check near-duplicate ratio and scene coverage. On no hits, first inspect index, feature model, and filter conditions before concluding data is truly missing.

Step 3: Decide reuse or targeted collection. If existing samples meet quality and coverage, form a specialized candidate set for human review, relabeling, and QA; if specific weather, lighting, or road conditions are absent, generate a directed collection list. Candidate sets must retain source, filtering rules, and data snapshot for reproducibility.

Step 4: Train and regression test. Training jobs read the fixed-version sample set, recording label version and model version. Before new model release, check overall metrics and also verify improvement on specialized scenes like rainy-night cut-in, while watching for regressions elsewhere. New bad cases re-enter the next round of data processing and retrieval.

The entire process makes raw objects, processing results, retrieval candidates, and model evaluations traceably connected. Teams ultimately answer four questions: Where did the problem occur? Where is the relevant data? Is the data sufficient? Was the improvement validated?

Minimum Acceptance Checklist Before Go-Live

Whether the solution works in production depends on indexes, permissions, quality, and resource configuration. Keep at least one machine-readable acceptance table:

Data Consistency – Paimon snapshot, Global Index version, processing model version, sample count.

Retrieval Quality – Labeled set, Recall@K, actual returned rows, false-positive and false-negative examples.

Query Performance – Cold/hot cache P50/P95/P99, successful QPS, concurrency tiers, Profile.

Resource Cost – Compute group specs, index size, OSS read volume, model invocation count, monthly cost.

Business Loop – Sample reuse rate, directed collection volume, time from bad case to usable training set.

If queries slow down, first use EXPLAIN to confirm top-K, scalar predicates, and full-text conditions follow the expected execution path, then check Profile for index search, candidate count, fetch reads, and cache. When fewer than K rows return, distinguish "insufficient after filtering", "index recall shortfall", and "candidate budget too small" – conflating them leads to completely different optimization directions.

Conclusion

The value of a multimodal data lake materializes the moment files are understood and reused for business. OSS stores raw content, DLF Paimon organizes evolvable data assets, and EMR Serverless StarRocks' Stella kernel plugs retrieval and analysis into a single SQL chain. For autonomous driving teams, this means a single model bad case can follow data relationships to historical samples, training sets, and the next evaluation round, letting settled data continuously participate in model iteration.