EMR Serverless StarRocks Supports Paimon 2.0 for Multimodal Analytics & Retrieval

Paimon 2.0: Multimodal Data Lake Scenarios and Technical Foundations

Traditional lakehouse analytics focused on structured business records. Multimodal applications now require raw media (video, images, audio, documents, sensor data) alongside orders, logs, tags, and metrics. Models extract semantics, enabling content-based sample discovery, correlation with business records, and feeding selected data into training and evaluation.

Public datasets illustrate the scale: Waymo Open Motion Dataset (Oct 2025) contains 103,354 20-second scene clips linking trajectories, maps, and image embeddings. Google DeepMind's Open X-Embodiment (2023) aggregates over 1 million task episodes across 22 robot types. In these datasets, an observation's meaning depends on temporal context, surroundings, and task goals, requiring simultaneous handling of content, relationships, and versions.

Typical Industry Scenarios

Autonomous Driving: Hard-case Retrieval & Incident Review – From a rainy-night driving anomaly, retrieve similar scenes, constrain by vehicle model and software version, correlate road conditions, point clouds, and vehicle state to diagnose common issues. Results form a verifiable, re-labelable sample set for model evaluation and training augmentation.

Embodied Intelligence: Task Retrieval & Policy Evaluation – From a robot insertion failure, retrieve similar visual states and action trajectories, constrain by task type and policy version, correlate contact forces and execution results to analyze failure causes. Results form replayable, comparable task samples for policy evaluation and data recollection.

Advertising: Creative Retrieval & Performance Analysis – From high-converting ads, retrieve creatives with similar composition, product, or style, constrain by region and campaign period, correlate impressions, clicks, and conversions to compare content feature performance. Results form a performance-tagged creative collection for iteration and testing.

Gaming: Asset Retrieval & Version Governance – From a character or scene design requirement, retrieve similar assets by reference image or text description, constrain by project and version, correlate licensing info and release records to verify reuse conditions. Results form a traceable asset inventory for R&D collaboration and asset reuse.

Multimodal data is large and frequently updated; low-cost storage and data consistency remain fundamental. Lakehouse keeps data in object storage, allowing different tasks to read from the same snapshot, reducing data movement and replica maintenance. Paimon 2.0 builds on real-time lake tables to retain raw media long-term while continuously backfilling tags and vectors, enabling a single table to support feature engineering, retrieval, analysis, and training.

Paimon 2.0 Core Capabilities and Implementation

The article focuses on multimodal Append tables with Row Tracking and Data Evolution enabled. Key capabilities:

Catalog & Table Metadata – Schema, Snapshot, manifest-list, manifest. Role: discover table, fix read version, determine visible files and indexes.

Data Evolution – Write updated columns to new files, reuse unchanged columns. Role: backfill features by column; at read time combine original and updated columns.

Row Tracking – _ROW_ID identifies record, _SEQUENCE_NUMBER identifies version. Role: align features with original samples, bridge index results to data location.

BLOB & VECTOR – Media, vectors, and regular columns organized separately. Role: read on demand, avoid loading irrelevant media content.

Global Index – Scalar, vector, full-text indexes and their coverage. Role: provide candidate row IDs or scores, support condition combination, merging, and fetch.

PyPaimon – Python read/write and column backfill, connect Ray, PyTorch. Role: let feature engineering, training data prep, and SQL analysis share the lake table.

Driving Frame Example: From Raw Media to Retrievable Features

A driving frame record incrementally gains features. Initial write stores capture time, vehicle model as regular columns; raw media stored independently via BLOB; Row Tracking assigns a row ID. After model computation, carrying the same row ID, Data Evolution backfills labels, evaluation results, and vectors: labels and scores go to regular columns, vectors use separately organized VECTOR files; unchanged business attributes and media files continue to be reused.

Labels and vectors can each build scalar and vector Global Indexes. After query selects a snapshot, Paimon uses that version's visible files and available indexes, combining column data on demand. Raw content and later-backfilled features remain in the same table, allowing StarRocks to perform retrieval, joins, and analysis via SQL.

StarRocks and Paimon 2.0: Unified Architecture for Analytics and Retrieval

From Delta Lake, Iceberg to Paimon, StarRocks has accumulated cross-format metadata management, SQL planning, distributed execution, and data caching. Multimodal lakehouse continues to leverage these: business analysis still needs joins, aggregations, sorts; object storage access requires I/O control; concurrent queries need resource management. Paimon 2.0's Global Index adds an index access path to query planning.

StarRocks' Paimon support covers vector and full-text search, scalar predicate pushdown, BLOB read, and cache integration. Queries can first narrow candidates via time, vehicle model, tags, then perform similarity or keyword search; hits then join business records for statistical analysis. Data stays in Paimon tables; retrieval and analysis share table version and query execution capabilities.

Global Index scalar conditions produce row ID sets, combinable via AND/OR; vector and full-text indexes also produce candidate scores for merging and ranking. Scalar candidate sets can serve as prefilter for vector search. StarRocks chooses column scan or index access based on query semantics and cost, arranging candidate merging and column data reads.

Both paths ultimately enter relational computation. Paimon manages table version, row identity, column files, and index organization; StarRocks handles SQL planning, parallel execution, caching, and resource management. A multi-table query can simultaneously use scan and index access; the actual order of Filter, Join, TopK, and late materialization is determined by the query plan.

From OLAP to Search: Query Execution Focus Shifts

OLAP: Narrow Column Scan Range via Data Skipping

Metadata pruning → Parallel column scan & filter → Join, aggregation, sort

Scan-heavy lake OLAP queries first use partition info, file metadata min/max statistics to skip files that cannot satisfy predicates. Readers further prune using row-group, page-level statistics. This data skipping reduces bytes read based on data range summaries; retained data still requires row-level filtering. Execution engine organizes parallel read tasks by file or intra-file ranges, reads columns on demand, batch decodes and filters, then feeds results into join, aggregation, sort operators. This path is common for reports, scenario statistics, and large-range correlation analysis; optimization focuses on reducing scanned bytes, improving batch read and compute throughput, and balancing work across nodes.

Search: Index Search First, Then Fetch by Sparse Row IDs

Index search → Candidate combination or ranking → Row ID location → On-demand fetch & downstream compute

Search workloads enter via indexes, finding candidate records per query conditions or vectors. Scalar retrieval typically outputs row ID sets; vector retrieval also outputs distances or scores; execution engine combines conditions, merges candidates, then fetches needed columns for relational compute. During fetch, Paimon locates relevant data files using row ID metadata in the query snapshot; readers read required columns. For Data Evolution tables, a single record's columns may come from multiple files, requiring combination by row identity and version. Sparse candidates narrow read targets; actual cost depends on hit file count, projected columns, and cache state.

StarRocks Execution Adaptations

Coordinated Scheduling of Search and Fetch – StarRocks' existing distributed execution and data caching provide parallel scheduling foundation. Vector and full-text indexes can be organized by index shard for search tasks; resulting row IDs locate files and columns to read. Index search and data read have different task boundaries, requiring separate parallelism tuning, combined with cache locality and node load scheduling. Fetch organizes batch requests by target files, reusing metadata and data caches to control object requests and cross-node transfer overhead.

Candidate Merging and Late Materialization – StarRocks' distributed merge and global late materialization framework first exchanges lightweight row IDs and scores, completing global candidate merge and ranking. Columns only needed for output (media URLs, scene descriptions) are batch-fetched after final TopK is determined, reducing useless reads, decoding, and cross-node transfers. Whether reads can be deferred depends on column usage: if filters or joins decide which records participate in ranking, those conditions must be processed before final TopK; relevant columns, and original vectors needed for precise ranking, may still be read early. Fetched data enters StarRocks' existing vectorized execution pipeline for downstream analysis. Example: after selecting top 100 most similar driving frames, join vehicle info, aggregate and sort by software version to analyze version distribution of hit samples.

Access Path Selection by Overall Cost – StarRocks' CBO uses statistics to estimate execution cost, working with predicate pushdown and join ordering to organize query plan. For Paimon index access, it must factor in index search volume, candidate scale, hit file distribution, and Data Evolution column merge cost, choosing access path together with downstream relational compute. When candidates are too many or too scattered, column scan may be cheaper; ANN queries need to compare latency and resource overhead under fixed recall targets.

Paimon 2.0 Best Practices: Data Management from Write to Query

Based on production experience, Paimon multimodal table management boils down to three things: control file size, backfill features orderly, maintain indexes timely. They respectively affect StarRocks' scan, column data combination, and index access cost. Observing query performance alongside background maintenance is key to diagnosing whether issues lie in access path or data organization itself.

Write & Compaction: Make File Layout Serve Queries

Design Partitioning with Write Batches – Partition by time, business domain, etc., to help StarRocks narrow read range; avoid over-partitioning and too-small batches. 1 TiB data written as 1 MiB files yields ~1.05 million objects; at 256 MiB yields ~4096 objects. Target file size must balance write latency, query pruning, and parallelism.

Calibrate Compaction Cadence with Actual Read Overhead – Continuously monitor file count, size distribution, compaction backlog, and correlate with StarRocks scan volume, object requests, cache hit rate to judge maintenance effectiveness. The Row Tracking tables discussed use unaware Append mode, so primary-key table bucket planning does not directly apply.

Data Evolution: Plan Backfill and Read Cost Together

Backfill by Row ID, Retain Feature Versions – Write only new or changed columns, record model version, compute time, processing state. Vectors from different models may not share feature space; queries must specify comparison scope. Row Tracking solves row correspondence; feature semantics and versioning remain a business concern.

Backfill Plan Includes Column File Compaction & Index Refresh – Partial updates accumulate column files; combine with StarRocks' commonly projected columns to evaluate combined read cost. When indexed columns update, check index coverage and version; maintenance involving row ID reassignment must also handle affected indexes and external references.

Global Index: Configure and Evaluate Around Real Queries

Equality, range, category filter – Typical index: BTree, Bitmap. Key focus: selectivity, cardinality, candidate count.

Text & keyword search – Typical index: Full-text index. Key focus: tokenization, relevance, search cost.

Vector similarity search – Typical index: IVF-PQ etc. vector indexes. Key focus: distance metric, index size, recall rate.

Balance Parallelism and Task Overhead with Index Shards – Too many vector/full-text index shards increase StarRocks scheduling and merge overhead; too large increases single-task load. BTree and Bitmap have their own ordering and range organization; cannot apply same partitioning strategy. Index shards ≠ table partitions or buckets.

Evaluate Full Query, Verify Index Coverage – Fix model, distance metric, recall target; compare cold/hot cache search, merge, fetch latency; adjust config based on candidate count and hit file distribution. After new data or feature updates, separately verify committed index coverage.

DLF Managed Maintenance: Align Maintenance State with Query Performance

Alibaba Cloud Data Lake Formation (DLF) can centrally manage Global Index async build, auto compaction, snapshot cleanup, partition lifecycle, and orphan file cleanup. Teams can evaluate index coverage, task backlog, resource consumption alongside StarRocks query latency, scan volume, cache performance to tune maintenance resources and frequency.

Data write success and index query readiness are two separate states to observe. DLF manages lake table and index maintenance; StarRocks plans scan or index access per query version; maintenance strategy must simultaneously satisfy retrieval timeliness, query performance, and cost requirements.

Ongoing Work: StarRocks Optimization Under Mixed Workloads

As Paimon multimodal tables carry more workloads, StarRocks must handle low-latency retrieval and complex OLAP queries concurrently, while coping with data changes and I/O contention from feature backfill, index build, and compaction. Future optimization focuses on resource isolation, index-aware query planning, and query–maintenance coordination, letting a single execution architecture better adapt to diverse loads.

Isolate Mixed Workloads via StarRocks Compute Groups & Scheduling – Interactive retrieval cares about tail latency; batch analytics cares about throughput. Different compute groups can serve these query types, each with separate concurrency, resource budgets, autoscaling policies. Further work will combine query characteristics to optimize task scheduling, cache reuse, hotspot handling, reducing high-throughput task interference on online retrieval. Beyond compute group isolation, StarRocks read pressure and background maintenance storage bandwidth must be jointly evaluated to fairly allocate shared I/O resources.

Make StarRocks Query Planning Explicitly Index-Coverage Aware – A time gap exists between data write completion and index readiness. StarRocks must organize access based on available indexes and their coverage in the query snapshot: when query semantics and reader capability allow, schedule supplemental reads for uncovered data; if business chooses to only search indexed data, explicitly define searchable scope. Future work will reflect this difference in query planning and performance observability, evaluating supplemental read cost, letting business explicitly choose among result completeness, data freshness, and query latency.

Use StarRocks Query Metrics for Lake Table Maintenance Decisions – Compaction and index refresh are executed by Paimon maintenance tasks or DLF; their effect ultimately shows in StarRocks scan, search, fetch overhead. Future work will link query latency, cache hit, read volume with file scale, index coverage, task backlog to guide maintenance frequency and resource allocation. StarRocks side needs improved metadata and cache management so new queries promptly use new data and index versions; running queries stay on fixed snapshot, coordinated with lake table version retention policy to guarantee read consistency.

Paimon 2.0 organizes raw media, continuously backfilled features, and indexes within a unified lake table and version system; Stella bridges index search, candidate merge, on-demand fetch, and downstream relational compute. A single query can start from similar frames or keywords, filter samples with business conditions, then explain results via joins and aggregations. Data stays in Paimon tables; feature engineering, retrieval analysis, and training data prep revolve around the same lake data.

On Alibaba Cloud, this pipeline is supported by lake table, maintenance service, and query engine together: Paimon organizes table versions, column data, indexes; DLF manages index build, compaction, etc.; Stella handles SQL planning, analytics, and retrieval execution. EMR Serverless StarRocks further provides fully managed instance management, elastic scaling, and multi-compute-group isolation, helping teams configure compute resources for interactive retrieval vs batch analytics, and adjust lake table maintenance strategies based on query performance.

Going forward, Stella will continue optimizing mixed-load isolation, index-aware query planning, and query–maintenance coordination, letting a single execution architecture better serve low-latency retrieval and complex analytics. Real workloads from Alibaba Group businesses and public cloud customers will keep feeding these efforts. As Paimon's data organization and indexing capabilities evolve, Stella will further expand multimodal analytics and retrieval on open lake tables.