StarRocks + Paimon Global Index: From File Scans to Row-Level Lookups with BTree & Vector Search
StarRocks now supports Paimon Global Index for fine-grained data access, enabling BTree indexes for high-selectivity scalar filters and IVF-PQ vector indexes for approximate nearest-neighbor search, with a two-phase execution architecture that combines structured filtering and semantic retrieval in a single SQL query.
What Is Paimon Global Index?
Paimon assigns a global Row ID to every row in an append-only table with Row Tracking and Data Evolution enabled. A Global Index maps business columns or vector embeddings to these Row IDs. Queries first search the index to obtain candidate Row IDs (and similarity scores for vector search), then locate and read only the corresponding data ranges, reducing I/O and computation.
The index is partitioned into Index Shards by partition and Row ID range, managed via Paimon Snapshot and Index Manifest. StarRocks distributes shard searches across multiple BE or CN nodes for parallelism.
Global Index vs. Partition Pruning and File Statistics
Partition pruning and file-level min/max statistics skip irrelevant files. Global Index goes further by providing candidate Row IDs for fine-grained range reads within the remaining files. Both optimizations can be combined: partition pruning first, then Global Index shard search, then Row ID–based data reads.
BTree Global Index: High-Selectivity Scalar Queries
BTree Global Index accelerates point lookups, IN predicates, and narrow range scans on high-cardinality columns such as order_id, user_id, tenant_id, product_code, device_id, price, rating, and high-cardinality categorical fields. The smaller the fraction of rows matched, the greater the benefit; if a predicate matches most rows, a full columnar scan may be faster.
Supported query conditions (index column on left, constant on right):
Equality (=)
IN (value list)
Range (>, >=, <, <=) — BETWEEN is rewritten to bounds
Prefix match via starts_with Complex OR and generic LIKE are not supported. The index returns a Row ID Bitmap representing the candidate row set.
Combining Multiple BTree Conditions
For a query filtering on tenant_id = 1001 AND price >= 3000 AND price < 8000 with indexes on both columns: tenant_id index returns Bitmap A price index returns Bitmap B
Intersect A and B to get candidate Row IDs
Read data for those Row IDs
This is a bitmap intersection, not a separate Bitmap Index type.
Vector Global Index: Semantic Top-K Retrieval
Vector Global Index uses Approximate Nearest Neighbor (ANN) search to recall candidate Row IDs and similarity scores from an embedding column, avoiding full-vector scans. Typical scenarios: RAG document retrieval, product semantic search, image search, recommendation candidate generation, similar-event lookup.
StarRocks supports these Paimon Vector Global Index types:
ivf-flativf-pq(IVF with Product Quantization)
Vector Query Functions
Use the matching distance function for the index metric: approx_l2_distance for L2 (Euclidean) distance approx_cosine_similarity for cosine similarity approx_inner_product for inner product
Critical constraints:
ANN returns approximate Top-K; results may differ from exact brute-force ranking.
Index distance metric must match the query function (e.g., cosine index → approx_cosine_similarity).
Combined BTree + Vector: Filter First, Then Semantic Search
Real-world vector searches often require structured filters (tenant, category, price). Running ANN first then filtering may leave too few valid results. The combined path:
BTree indexes produce a Row ID Bitmap for business predicates.
Bitmap is passed as ANN Pre-filter to the Vector Index.
Vector Index performs ANN search only within the pre-filtered candidate set.
Returns candidate Row IDs and scores.
Read candidate data, apply remaining filters/projections, produce final Top-K.
All expressed in a single SQL; the optimizer automatically chooses the joint path when indexes exist and cover the required Row ID range.
Example query:
SELECT
product_id,
title,
price,
approx_cosine_similarity(
CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
embedding
) AS score
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
AND category = 'camera'
AND price >= 3000
AND price < 8000
ORDER BY score DESC
LIMIT 20;Note: the 4-dimensional vector is for illustration; actual queries must match the index dimension (e.g., 768).
Two-Phase Execution Architecture
StarRocks separates index search from data reading:
Phase 1: Distributed Index Shard Search
FE Optimizer reads Paimon Snapshot and Index Manifest, identifies usable BTree/Vector indexes, determines Index Shards to search, and decides which BTree predicates serve as ANN Pre-filters.
FE assigns shards to multiple BE/CN nodes for parallel search.
BTree shards return Row ID Bitmaps; Vector shards return local candidate Row IDs + scores.
Results are merged: Bitmaps are unioned; vector candidates are globally merged for Top-K.
Phase 2: Row ID–Based Data Read
Merged index results are sent to Paimon, which generates IndexedSplits — essentially DataSplits augmented with Row Ranges and optional vector scores.
Paimon Native Reader reads only the specified row ranges.
StarRocks completes remaining filters, projections, joins, aggregations, and final Top-K.
Both phases operate on the same Paimon Snapshot, guaranteeing consistent data versions. Index search and data reading scale independently.
Usage Guide: End-to-End Example
1. Create Table and Indexes in Paimon (Spark/Flink)
CREATE TABLE demo.product_search (
product_id BIGINT,
tenant_id BIGINT,
category STRING,
price DOUBLE,
title STRING,
embedding ARRAY<FLOAT>
) TBLPROPERTIES (
'bucket' = '-1',
'row-tracking.enabled' = 'true',
'data-evolution.enabled' = 'true',
'global-index.enabled' = 'true'
);Create BTree indexes:
CALL sys.create_global_index(table => 'demo.product_search', index_column => 'tenant_id', index_type => 'btree');
CALL sys.create_global_index(table => 'demo.product_search', index_column => 'category', index_type => 'btree');
CALL sys.create_global_index(table => 'demo.product_search', index_column => 'price', index_type => 'btree');Create IVF-PQ vector index:
CALL sys.create_global_index(
table => 'demo.product_search',
index_column => 'embedding',
index_type => 'ivf-pq',
options => 'ivf-pq.dimension=768,
ivf-pq.distance.metric=cosine,
ivf-pq.nlist=256,
ivf-pq.pq.m=16'
);Key parameters: dimension must match embedding size; distance.metric must match query function; nlist = IVF cluster count; pq.m = PQ segments (dimension must be divisible by m). Index creation and incremental maintenance are handled by Paimon; StarRocks discovers and uses existing indexes.
2. Create DLF REST Catalog in StarRocks
CREATE EXTERNAL CATALOG paimon_dlf
PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "rest",
"uri" = "https://cn-hangzhou-vpc.dlf.aliyuncs.com",
"paimon.catalog.warehouse" = "<dlf_catalog_name>",
"token.provider" = "dlf"
);urimust be the DLF regional REST endpoint; warehouse is the DLF Catalog instance name, not an OSS/HDFS path.
3. Enable Two-Phase Global Index Scan
SET paimon_global_index_scan_stage = 2;Default in target versions; values control fallback behavior.
4. Query BTree Global Index
SELECT product_id, title, category, price
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
AND price >= 3000
AND price < 8000;Optimizer intersects Bitmaps from tenant_id and price indexes.
5. Query IVF Vector Global Index
SET enable_paimon_vector_index = true;
SET ann_params = '{"ivf.nprobe":"32"}';SELECT
product_id,
title,
category,
price,
approx_cosine_similarity(
CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
embedding
) AS score
FROM paimon_dlf.demo.product_search
ORDER BY score DESC
LIMIT 20;Vector query requirements:
Query vector is a constant ARRAY<FLOAT>.
Vector dimension matches index.
Query function matches index metric.
Only one vector ordering expression.
Must include LIMIT.
L2 uses ASC; Cosine and Inner Product use DESC.
For combined structured + vector queries, use the joint SQL shown earlier; optimizer decides on the combined path.
Summary
Partition pruning and file statistics skip files; Paimon Global Index uses Row IDs to further narrow candidate ranges. BTree Global Index serves high-selectivity scalar filters (point, IN, narrow range). Vector Global Index (IVF-Flat, IVF-PQ) serves ANN semantic search. Combined, they enable filter-then-vector retrieval in one SQL. The two-phase architecture parallelizes index shard searches across BE/CN nodes, then uses Paimon IndexedSplits for precise row-range reads via the Native Reader. Users write ordinary SQL; the optimizer handles index selection, bitmap intersection, ANN pre-filtering, and merge logic transparently.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
