EMR Serverless StarRocks: Paimon Global Index Enables Row-Level and Vector Search
This article explains how EMR Serverless StarRocks leverages Paimon Global Index, including BTree for high-selectivity scalar filters and vector indexes for approximate nearest neighbor search, detailing the two-phase query architecture, index creation, SQL usage, and combined scalar-vector query execution.
What Is Paimon Global Index?
Paimon assigns a global Row ID to each row in a table. Global Index builds a mapping from business columns or vectors to Row IDs. Queries first search the index to obtain candidate Row IDs, then locate and read only the corresponding data ranges, reducing the amount of data scanned and processed.
This capability targets Append Tables with Row Tracking and Data Evolution enabled. Table creation and index configuration are shown in the usage examples later.
Query Process
Search Global Index by filter conditions or query vector.
Obtain candidate Row IDs; vector search also returns a similarity Score.
Locate data ranges by Row ID and read candidate data.
Execute remaining filters, projections, and Top-K operations.
The term "Global" refers to table-level Row ID mapping, not a single monolithic index file. Physically, Global Index is split into multiple Index Shards by partition and Row ID range, managed by Paimon Snapshot and Index Manifest. EMR Serverless StarRocks distributes these shards across multiple BE or CN nodes for parallel search.
Global Index vs. Partition/File-Level Pruning
Location Granularity : Partition & File-Level Pruning uses partition, file, or Row Group; Paimon Global Index uses table-level Row ID.
Query Result : Partition & File-Level Pruning returns candidate data files; Paimon Global Index returns candidate Row IDs.
Primary Goal : Partition & File-Level Pruning skips irrelevant files; Paimon Global Index directly locates target data rows.
Typical Scenarios : Partition & File-Level Pruning suits partition queries and wide-range analytics; Paimon Global Index suits point lookups, narrow-range filters, and vector Top-K.
Vector Search : Partition & File-Level Pruning is not applicable; Paimon Global Index returns Row ID and Score.
These optimizations work together: a query can first prune partitions, then search relevant Global Index Shards, and finally read data by candidate Row IDs.
BTree Global Index: Accelerating High-Selectivity Scalar Queries
Queries that look up a single order by ID or filter products by tenant and price range typically return few rows. BTree Global Index finds matching Row IDs first, then reads only those rows, reducing scan and filter overhead.
Suitable for basic numeric and string columns frequently appearing in WHERE clauses, such as:
Order ID, User ID, Tenant ID
Product codes, device numbers
Price, rating, and other numeric fields
High-cardinality business category fields
The key factor is selectivity: the smaller the fraction of rows matching the condition, the more benefit BTree Global Index provides. If a condition matches most rows, a regular columnar scan may be better.
BTree Global Index returns a Row ID Bitmap representing the candidate row set; subsequent steps read business data using these Row IDs.
Supported Query Conditions
Equality & Inequality : tenant_id = 1001, status != 'deleted' Range : price >= 3000 AND price < 8000 IN / NOT IN : product_id IN (101, 205, 309) Null Check : category IS NOT NULL String Prefix : starts_with(product_code, 'ALI-') BETWEEN can be rewritten as lower/upper bounds, so range indexes apply. For optimizer recognition, write conditions as "indexed column on left, constant on right". Complex OR and regular LIKE are not yet supported; use starts_with for prefix matching.
Combining Multiple BTree Conditions
Example: WHERE tenant_id = 1001 AND price >= 3000 AND price < 8000 When both tenant_id and price have usable BTree Global Indexes, StarRocks queries each index separately and merges results: tenant_id index returns Row ID Bitmap A. price index returns Row ID Bitmap B.
Intersect the two Bitmaps to get candidate Row IDs satisfying both conditions.
Read data for those Row IDs.
The Row ID Bitmap is the result of BTree index query, not a separate Bitmap Index. For point lookups, IN queries, and narrow range queries, this path reduces reading of rows that would later be filtered out.
Vector Global Index: Semantic Top-K Retrieval
In product semantic search, RAG document retrieval, or image search, queries need the most similar results. Text, documents, images, and audio/video are converted to embeddings; vector distance or similarity drives retrieval.
Scanning all vectors and computing distances per query scales poorly. Vector Global Index uses Approximate Nearest Neighbor (ANN) search to first recall candidate Row IDs and their distance/similarity scores, then reads only those candidates.
Typical Scenarios
RAG document chunk recall
Product semantic search
Image search
Recommendation candidate recall
Similar event or scene retrieval
Supported Paimon Vector Global Index Types
diskann : Disk-based Vamana graph with PQ compression and raw vector re-ranking. Applicable for large-scale vector libraries with lower memory footprint.
ivf-flat : IVF partitioning with raw vector storage. Applicable when higher recall priority is needed and larger index size is acceptable.
ivf-pq : IVF with Product Quantization compression. Applicable when focusing on index size and query resource consumption.
ivf-sq : IVF with Scalar Quantization compression. Applicable for balanced compressed storage, recall, and query performance.
ivf-rq : IVF with Rotated Residual Quantization compression. Applicable for high-dimensional vectors, trading off between index size and recall.
EMR Serverless StarRocks Vector Query Functions
approx_l2_distance : Sort direction ASC. Smaller L2 distance means more similar.
approx_cosine_similarity : Sort direction DESC. Larger cosine similarity means more similar.
approx_inner_product : Sort direction DESC. Larger inner product means more similar.
Two important notes:
ANN returns approximate Top-K results, not guaranteed identical to exact full-scan ranking.
Index distance metric must match the SQL query function. For example, a cosine-metric index requires approx_cosine_similarity.
BTree + Vector: Filter Business Scope First, Then Semantic Search
Real-world vector search often requires structured filters (e.g., tenant, category, price). Running ANN first then filtering may leave too few valid results.
BTree and Vector Global Indexes can combine: BTree finds Row IDs matching business conditions, then that Bitmap serves as an ANN pre-filter for the vector index, restricting the search space.
Combined Retrieval Flow
BTree generates Row ID Bitmap from tenant, category, price conditions.
Bitmap passed as ANN pre-filter to Vector Index.
Vector Index performs ANN search within the business-filtered candidate set.
Returns candidate Row IDs and Scores.
Read candidate data, execute remaining processing, produce final Top-K.
Users write ordinary SQL; no explicit index hints needed. Example query restricting tenant, camera category, price range, then ordering by cosine similarity and limiting to 20:
SELECT
product_id,
title,
price,
approx_cosine_similarity(
CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
embedding
) AS score
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
AND category = 'camera'
AND price >= 3000
AND price < 8000
ORDER BY score DESC
LIMIT 20;The 4-dimensional vector is for illustration only; actual queries must match the embedding dimension (e.g., 768). When both business filter columns and vector column have usable indexes covering the required Row ID range, the optimizer automatically chooses the combined BTree+Vector path.
Two-Phase Query Execution Architecture
EMR Serverless StarRocks uses a two-phase execution: Phase 1 parallel index search to get candidate Row IDs; Phase 2 reads data by those Row IDs and completes further computation.
FE Identifies Index Conditions
The FE Optimizer obtains the Paimon Snapshot and Index Manifest, determines:
Which filter conditions can use BTree Index.
Whether Top-K expression can use Vector Index.
Which Index Shards to search.
Which BTree conditions can serve as ANN pre-filter.
Whether each index fully covers the query's required Row ID range.
Index search and data reading use the same Paimon Snapshot, ensuring consistent data versions across phases.
Phase 1: Distributed Index Shard Search
FE distributes Index Shards to multiple BE/CN nodes for parallel execution:
BTree returns matching Row ID Bitmap.
Vector Index returns candidate Row IDs and Scores.
In combined queries, BTree Bitmap acts as ANN pre-filter.
Results are aggregated: BTree Bitmaps merged across shards; vector results merged globally to form the candidate set for Phase 2.
Phase 2: Read Data by Row ID
StarRocks passes aggregated index results to Paimon. Paimon generates IndexedSplit based on Row ID coverage of data files. IndexedSplit extends DataSplit with required Row Ranges and optional vector Scores.
Paimon Native Reader reads candidate data by specified ranges; StarRocks then executes remaining filters, projections, joins, aggregations, and final Top-K.
This architecture reduces reads of non-matching data; index search and data reading can scale independently.
Usage Guide: Querying Paimon Global Index with EMR Serverless StarRocks
Example: product semantic search table.
Prepare Table and Indexes in Paimon
Paimon Global Index targets Append Tables without primary keys. Table must enable Row Tracking, Data Evolution, and dynamic Bucket.
Execute the following in Paimon's Spark or Flink environment:
CREATE TABLE demo.product_search (
product_id BIGINT,
tenant_id BIGINT,
category STRING,
price DOUBLE,
title STRING,
embedding ARRAY<FLOAT>
)
TBLPROPERTIES (
'bucket' = '-1',
'row-tracking.enabled' = 'true',
'data-evolution.enabled' = 'true',
'global-index.enabled' = 'true'
);Create BTree Global Indexes for tenant, category, price:
CALL sys.create_global_index(
table => 'demo.product_search',
index_column => 'tenant_id',
index_type => 'btree'
);
CALL sys.create_global_index(
table => 'demo.product_search',
index_column => 'category',
index_type => 'btree'
);
CALL sys.create_global_index(
table => 'demo.product_search',
index_column => 'price',
index_type => 'btree'
);Create IVF-PQ Vector Global Index for embedding:
CALL sys.create_global_index(
table => 'demo.product_search',
index_column => 'embedding',
index_type => 'ivf-pq',
options => '
ivf-pq.dimension=768,
ivf-pq.distance.metric=cosine,
ivf-pq.nlist=256,
ivf-pq.pq.m=16'
);Key parameters: dimension: vector dimension, must match actual embedding dimension. distance.metric: distance metric, must match query function. nlist: number of IVF clusters. pq.m: number of PQ segments; vector dimension must be divisible by this value.
Index creation and incremental maintenance are handled by Paimon; StarRocks discovers and uses existing indexes.
Create DLF REST Catalog in EMR Serverless StarRocks
After binding a RAM user with DLF permissions, create catalog without AccessKey/Secret:
CREATE EXTERNAL CATALOG paimon_dlf
PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "rest",
"uri" = "https://cn-hangzhou-vpc.dlf.aliyuncs.com",
"paimon.catalog.warehouse" = "<dlf_catalog_name>",
"token.provider" = "dlf"
);Notes: uri must be the actual DLF REST endpoint for the region; paimon.catalog.warehouse is the DLF Catalog instance name, not an OSS/HDFS path.
Enable Two-Phase Global Index Query
Target versions default to two-phase path; can be set explicitly per session: SET paimon_global_index_scan_stage = 2; Values:
0 : No Global Index, regular data scan.
1 : Paimon SDK single-phase query path.
2 : EMR Serverless StarRocks BE distributed two-phase query path.
Query BTree Global Index
Example filtering tenant 1001 and price in [3000, 8000):
SELECT
product_id,
title,
category,
price
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
AND price >= 3000
AND price < 8000;Optimizer detects usable BTree indexes on tenant_id and price, intersects their Row ID Bitmaps, then reads matching rows.
Query IVF Vector Global Index
Enable vector index rewrite: SET enable_paimon_vector_index = true; Adjust IVF search scope via ann_params: SET ann_params = '{"ivf.nprobe":"32"}'; Example Top-20 by cosine similarity:
SELECT
product_id,
title,
category,
price,
approx_cosine_similarity(
CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
embedding
) AS score
FROM paimon_dlf.demo.product_search
ORDER BY score DESC
LIMIT 20;The 4-dim vector is illustrative; replace with 768-dim query vector matching the index. Vector index usage requires:
Query vector is a constant ARRAY<FLOAT>.
Query vector dimension matches index.
Query function matches index distance metric.
Only one vector ordering expression.
Query includes LIMIT.
L2 distance uses ASC; Cosine and Inner Product use DESC.
For combined business filters and vector search, use the joint query SQL from section 4; optimizer decides whether to use BTree+Vector path.
Summary
Partition pruning and file-level statistics skip irrelevant files; Paimon Global Index further narrows candidate data via Row ID, enabling StarRocks to search indexes first, then read only matching data.
BTree Global Index suits high-selectivity point lookups, IN queries, and narrow range filters. Vector Global Index targets RAG, semantic product search, image retrieval, and recommendation recall. Combined, they first restrict business scope (tenant, category, price) then perform vector search.
The two-phase architecture parallelizes index shard search across BE/CN nodes to obtain candidate Row IDs and optional Scores; Paimon generates IndexedSplit for Native Reader to fetch candidate data.
Users write standard SQL, achieving structured filtering, semantic search, and subsequent joins, aggregations, and Top-K in a single Paimon table.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
