Skip to content
StarRocks + Paimon Global Index: From File Scans to Row-Level Lookups with BTree & Vector Search
Databases 20 min read

StarRocks + Paimon Global Index: From File Scans to Row-Level Lookups with BTree & Vector Search

StarRocks now supports Paimon Global Index for fine-grained data access, enabling BTree indexes for high-selectivity scalar filters and IVF-PQ vector indexes for approximate nearest-neighbor search, with a two-phase execution architecture that combines structured filtering and semantic retrieval in a single SQL query.

StarRocks
StarRocks
StarRocks

What Is Paimon Global Index?

Paimon assigns a global Row ID to every row in an append-only table with Row Tracking and Data Evolution enabled. A Global Index maps business columns or vector embeddings to these Row IDs. Queries first search the index to obtain candidate Row IDs (and similarity scores for vector search), then locate and read only the corresponding data ranges, reducing I/O and computation.

The index is partitioned into Index Shards by partition and Row ID range, managed via Paimon Snapshot and Index Manifest. StarRocks distributes shard searches across multiple BE or CN nodes for parallelism.

Global Index vs. Partition Pruning and File Statistics

Partition pruning and file-level min/max statistics skip irrelevant files. Global Index goes further by providing candidate Row IDs for fine-grained range reads within the remaining files. Both optimizations can be combined: partition pruning first, then Global Index shard search, then Row ID–based data reads.

BTree Global Index: High-Selectivity Scalar Queries

BTree Global Index accelerates point lookups, IN predicates, and narrow range scans on high-cardinality columns such as order_id, user_id, tenant_id, product_code, device_id, price, rating, and high-cardinality categorical fields. The smaller the fraction of rows matched, the greater the benefit; if a predicate matches most rows, a full columnar scan may be faster.

Supported query conditions (index column on left, constant on right):

Equality (=)

IN (value list)

Range (>, >=, <, <=) — BETWEEN is rewritten to bounds

Prefix match via starts_with Complex OR and generic LIKE are not supported. The index returns a Row ID Bitmap representing the candidate row set.

Combining Multiple BTree Conditions

For a query filtering on tenant_id = 1001 AND price >= 3000 AND price < 8000 with indexes on both columns: tenant_id index returns Bitmap A price index returns Bitmap B

Intersect A and B to get candidate Row IDs

Read data for those Row IDs

This is a bitmap intersection, not a separate Bitmap Index type.

Vector Global Index: Semantic Top-K Retrieval

Vector Global Index uses Approximate Nearest Neighbor (ANN) search to recall candidate Row IDs and similarity scores from an embedding column, avoiding full-vector scans. Typical scenarios: RAG document retrieval, product semantic search, image search, recommendation candidate generation, similar-event lookup.

StarRocks supports these Paimon Vector Global Index types:

ivf-flat
ivf-pq

(IVF with Product Quantization)

Vector Query Functions

Use the matching distance function for the index metric: approx_l2_distance for L2 (Euclidean) distance approx_cosine_similarity for cosine similarity approx_inner_product for inner product

Critical constraints:

ANN returns approximate Top-K; results may differ from exact brute-force ranking.

Index distance metric must match the query function (e.g., cosine index → approx_cosine_similarity).

Combined BTree + Vector: Filter First, Then Semantic Search

Real-world vector searches often require structured filters (tenant, category, price). Running ANN first then filtering may leave too few valid results. The combined path:

BTree indexes produce a Row ID Bitmap for business predicates.

Bitmap is passed as ANN Pre-filter to the Vector Index.

Vector Index performs ANN search only within the pre-filtered candidate set.

Returns candidate Row IDs and scores.

Read candidate data, apply remaining filters/projections, produce final Top-K.

All expressed in a single SQL; the optimizer automatically chooses the joint path when indexes exist and cover the required Row ID range.

Example query:

SELECT
  product_id,
  title,
  price,
  approx_cosine_similarity(
    CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
    embedding
  ) AS score
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
  AND category = 'camera'
  AND price >= 3000
  AND price < 8000
ORDER BY score DESC
LIMIT 20;

Note: the 4-dimensional vector is for illustration; actual queries must match the index dimension (e.g., 768).

Two-Phase Execution Architecture

StarRocks separates index search from data reading:

Phase 1: Distributed Index Shard Search

FE Optimizer reads Paimon Snapshot and Index Manifest, identifies usable BTree/Vector indexes, determines Index Shards to search, and decides which BTree predicates serve as ANN Pre-filters.

FE assigns shards to multiple BE/CN nodes for parallel search.

BTree shards return Row ID Bitmaps; Vector shards return local candidate Row IDs + scores.

Results are merged: Bitmaps are unioned; vector candidates are globally merged for Top-K.

Phase 2: Row ID–Based Data Read

Merged index results are sent to Paimon, which generates IndexedSplits — essentially DataSplits augmented with Row Ranges and optional vector scores.

Paimon Native Reader reads only the specified row ranges.

StarRocks completes remaining filters, projections, joins, aggregations, and final Top-K.

Both phases operate on the same Paimon Snapshot, guaranteeing consistent data versions. Index search and data reading scale independently.

Usage Guide: End-to-End Example

1. Create Table and Indexes in Paimon (Spark/Flink)

CREATE TABLE demo.product_search (
  product_id BIGINT,
  tenant_id  BIGINT,
  category   STRING,
  price      DOUBLE,
  title      STRING,
  embedding  ARRAY<FLOAT>
) TBLPROPERTIES (
  'bucket' = '-1',
  'row-tracking.enabled' = 'true',
  'data-evolution.enabled' = 'true',
  'global-index.enabled' = 'true'
);

Create BTree indexes:

CALL sys.create_global_index(table => 'demo.product_search', index_column => 'tenant_id', index_type => 'btree');
CALL sys.create_global_index(table => 'demo.product_search', index_column => 'category', index_type => 'btree');
CALL sys.create_global_index(table => 'demo.product_search', index_column => 'price', index_type => 'btree');

Create IVF-PQ vector index:

CALL sys.create_global_index(
  table => 'demo.product_search',
  index_column => 'embedding',
  index_type => 'ivf-pq',
  options => 'ivf-pq.dimension=768,
              ivf-pq.distance.metric=cosine,
              ivf-pq.nlist=256,
              ivf-pq.pq.m=16'
);

Key parameters: dimension must match embedding size; distance.metric must match query function; nlist = IVF cluster count; pq.m = PQ segments (dimension must be divisible by m). Index creation and incremental maintenance are handled by Paimon; StarRocks discovers and uses existing indexes.

2. Create DLF REST Catalog in StarRocks

CREATE EXTERNAL CATALOG paimon_dlf
PROPERTIES (
  "type" = "paimon",
  "paimon.catalog.type" = "rest",
  "uri" = "https://cn-hangzhou-vpc.dlf.aliyuncs.com",
  "paimon.catalog.warehouse" = "<dlf_catalog_name>",
  "token.provider" = "dlf"
);
uri

must be the DLF regional REST endpoint; warehouse is the DLF Catalog instance name, not an OSS/HDFS path.

3. Enable Two-Phase Global Index Scan

SET paimon_global_index_scan_stage = 2;

Default in target versions; values control fallback behavior.

4. Query BTree Global Index

SELECT product_id, title, category, price
FROM paimon_dlf.demo.product_search
WHERE tenant_id = 1001
  AND price >= 3000
  AND price < 8000;

Optimizer intersects Bitmaps from tenant_id and price indexes.

5. Query IVF Vector Global Index

SET enable_paimon_vector_index = true;
SET ann_params = '{"ivf.nprobe":"32"}';
SELECT
  product_id,
  title,
  category,
  price,
  approx_cosine_similarity(
    CAST([0.12, -0.03, 0.27, 0.08] AS ARRAY<FLOAT>),
    embedding
  ) AS score
FROM paimon_dlf.demo.product_search
ORDER BY score DESC
LIMIT 20;

Vector query requirements:

Query vector is a constant ARRAY<FLOAT>.

Vector dimension matches index.

Query function matches index metric.

Only one vector ordering expression.

Must include LIMIT.

L2 uses ASC; Cosine and Inner Product use DESC.

For combined structured + vector queries, use the joint SQL shown earlier; optimizer decides on the combined path.

Summary

Partition pruning and file statistics skip files; Paimon Global Index uses Row IDs to further narrow candidate ranges. BTree Global Index serves high-selectivity scalar filters (point, IN, narrow range). Vector Global Index (IVF-Flat, IVF-PQ) serves ANN semantic search. Combined, they enable filter-then-vector retrieval in one SQL. The two-phase architecture parallelizes index shard searches across BE/CN nodes, then uses Paimon IndexedSplits for precise row-range reads via the Native Reader. Users write ordinary SQL; the optimizer handles index selection, bitmap intersection, ANN pre-filtering, and merge logic transparently.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

StarRocksPaimonlakehouseGlobal IndexANN SearchVector IndexIVF-PQBTree Index
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.