StarRocks + Paimon: Bitmap Index Deep Dive with Official Standards & Production Cases

Background and Architecture

When StarRocks accesses Paimon tables via the Paimon Catalog, it pushes WHERE filter predicates down to Paimon. Paimon first uses file-level indexes (File Index) for data pruning and row filtering, returning only filtered data to StarRocks for aggregation and sorting. The index effectiveness and performance ceiling are entirely determined by Paimon's underlying File Index architecture.

Paimon File Index Official Core Definitions

Index Creation Rule: Configuring file-index.${index_type}.columns causes Paimon to generate an independent index file for each data file, creating a 1:1 strong binding between data and index files.

Index File Storage: Small indexes are embedded directly into Manifest metadata files; larger indexes are stored in the same directory as their corresponding data files.

Composite Index Capability: A single index file can simultaneously carry multiple columns and multiple index types (BloomFilter + Bitmap + Range-Bitmap coexistence).

Unified Index File Binary Format (Official Standard)

All File Indexes share a common file header structure divided into HEAD (metadata) and BODY (index data) , encoded in BIG_ENDIAN. This strong binding means writes, compaction, and data updates all trigger index rebuilds — the core reason index count must be limited.

Three File Index Types: Capability Differences

(1) BloomFilter Index

Config: file-index.bloom-filter.columns, fpp (false positive probability), items (estimated unique values per file)

Structure: 4-byte hash function count + bloom filter bitstream; strings/binary use XX Hash, numerics use dedicated numeric hash.

Core Capability: File-level existence check only , no row-level filtering. Preferred for high-cardinality fields.

Partition Requirement: Must pair with partition pruning; without partition conditions, full table scan degrades performance severely.

(2) Bitmap Index (V2 Mainstream)

Config: file-index.bitmap.columns, index-block-size (secondary index block, default 16KB)

Structure: Version header → total row count → null marker → index block map → serialized RoaringBitmap data.

Supported Types: TinyInt/SmallInt/Int/BigInt/Date/Time/Timestamp/Char/Varchar/String/Boolean

Core Capability: Row-level precise equality/IN filtering , bitmap intersection/union operations highly efficient. Optimal for low-cardinality fields.

Partition Requirement: Must pair with partition pruning; pure Bitmap scenarios require partition as mandatory prerequisite.

(3) Range-Bitmap Index

Config: file-index.range-bitmap.columns, chunk-size (data block, default 16KB)

Advantage: Smaller than regular Bitmap, supports equality, range, AND/OR, TOPN queries; fits medium-cardinality scenarios.

Disadvantage: Pure equality query performance slightly lower than regular Bitmap.

Extended Types: Adds Double/Float support on top of Bitmap.

Limitation: TOPN optimization only works for Append-Only tables; Spark engine support more mature.

Official Special Rule: Paimon docs explicitly state: partition key is not necessary for Range-Bitmap query optimization . Its built-in global dictionary and bit-slice index natively support cross-file, cross-partition range computation without relying on partition pruning — the core difference from Bitmap/BloomFilter.

Bitmap Index Official Applicable Scenarios & SQL Field Identification

Applicable Query Patterns

Single-field equality (highest priority): WHERE status = 1 IN collection queries (high-frequency business): WHERE city_id IN (1001,1002,1003) Multi-field equality combination (bitmap intersection): WHERE status=2 AND channel=5 AND is_valid=true Group aggregation, COUNT(DISTINCT): SELECT status, COUNT(*) FROM paimon_tbl GROUP BY status /

SELECT COUNT(DISTINCT status) FROM paimon_tbl

Explicitly Inapplicable Scenarios (Bitmap Completely Ineffective)

Range queries: > / < / >= / <= / BETWEEN Fuzzy queries: LIKE, regex matching

Sorting, TOPN queries (prefer Range-Bitmap)

Full-table large-range queries without partition conditions (Bitmap performance extremely poor)

Four-Dimensional Field Suitability Assessment (Production Standard)

Judgment based on business SQL, field characteristics, data distribution without parsing underlying files. The article includes a decision matrix image (not reproduced here) combining official cardinality specs and industry practice.

Practical Field Classification Examples

✅ Recommended for Bitmap (low cardinality, equality queries): status, pay_type, channel, is_valid, area_id.

❌ Prohibited for Bitmap (high cardinality / range queries): order_id, user_id (extreme cardinality), amount (range queries), create_time (time ranges).

Bitmap Index Count Thresholds: Optimal, Warning, Red Line

Based on File Index 1:1 binding to data files , index count directly impacts write performance, compaction duration, storage overhead, optimizer efficiency . Two standards distinguished: Append-Only regular tables and Primary Key tables (LSM-Tree based, updates trigger index rebuild, stricter constraints).

Count Threshold Specification

The article includes a threshold table image. Key points: thresholds apply simultaneously to Bitmap and Range-Bitmap ; combined index totals must also respect limits.

Harm of Excessive Bitmap Indexes

Storage Bloat: Each Bitmap index stored independently, single index adds 5%~20% storage; multiple indexes double cost.

Write Latency Increase: Flushing data files requires generating bitmap data for each column; more indexes = longer file close time.

Compaction Pressure Surge: Merging small files requires full rebuild of all Bitmap indexes for new data files; compaction time grows linearly with index count.

Optimizer Burden: Excessive indexes increase predicate matching and bitmap operation overhead; query benefit diminishes or reverses.

Production Incident Cases

Autonomous Driving Data Platform: Primary key table with 12 Bitmap indexes; high-frequency updates caused compaction timeouts, small file accumulation, query latency spiked from 200ms to 800ms; reducing to 3 core indexes restored performance.

E-commerce Order Database: Append-Only log table with 9 Bitmap indexes; storage bloated 28%, nightly batch write duration increased 40%; cutting to 4 indexes stabilized.

Index Replacement Rules: Alternatives for Bitmap Over-limit / Mismatch

When field cardinality exceeds threshold, query type mismatches, index count hits red line , replace sequentially with Range-Bitmap, BloomFilter per Paimon official capability boundaries.

3.1 Replacement by Field Cardinality (Core Rule)

Decision matrix image provided.

3.2 Replacement by Query Type (Scenario Rule)

Decision matrix image provided.

3.3 Replacement Configurations + Official SQL Examples

-- 1. Medium cardinality + range query: Bitmap → Range-Bitmap
ALTER TABLE paimon_tbl SET (
  'file-index.range-bitmap.columns' = 'score,price',
  'file-index.range-bitmap.score.chunk-size' = '16384' -- default 16KB
);

-- 2. High cardinality unique ID: Bitmap → BloomFilter
ALTER TABLE paimon_tbl SET (
  'file-index.bloom-filter.columns' = 'order_id',
  'file-index.bloom-filter.order_id.fpp' = '0.03',
  'file-index.bloom-filter.order_id.items' = '100000'
);

-- 3. Extreme cardinality point lookup: add B-Tree global index
ALTER TABLE paimon_tbl SET (
  'global-index.enabled' = 'true',
  'global-index.type' = 'b-tree',
  'global-index.columns' = 'user_id'
);

Official Native Examples (Range-Bitmap without partition, partition key dt omitted)

-- Official Example 1: Equality query, no partition condition (Range-Bitmap works)
SELECT * FROM TABLE WHERE score = 100;

-- Official Example 2: IN query, no partition condition
SELECT * FROM TABLE WHERE score IN (60, 80);

-- Official Example 3: Range query, no partition condition
SELECT * FROM TABLE WHERE score > 60;

-- Official Example 4: AND/OR composite, no partition condition
SELECT * FROM TABLE WHERE class_id = 1 AND score < 60 OR score > 80;

Layered Performance Optimization: Partition Pruning + BloomFilter + Bitmap Golden Combination

Combining Paimon storage architecture and StarRocks pushdown logic, two optimization logics distinguished:

Bitmap + BloomFilter: Must rely on partition pruning; three-layer filtering is optimal.

Range-Bitmap: Functionally partition pruning not required (official definition), can filter independently; but production large-data scenarios still recommend adding partition to further reduce scan range.

4.1 Two Execution Chains

Chain 1: Bitmap + BloomFilter (Traditional Combo, Strong Partition Dependency)

StarRocks pushes query SQL
  ↓
Layer 1: Partition pruning (mandatory) → filter 90%+ irrelevant partitions
  ↓
Layer 2: BloomFilter index (file-level) → exclude data files not containing target values
  ↓
Layer 3: Bitmap index (row-level) → precisely filter invalid rows
  ↓
Final dataset returned to StarRocks for computation

Chain 2: Range-Bitmap (Independent Mode, Partition Optional)

StarRocks pushes query SQL
  ↓
[Optional] Partition pruning → filter irrelevant partitions as needed (works without)
  ↓
Range-Bitmap index (file+row integrated filtering) → complete equality/range/composite filtering
  ↓
Final dataset returned to StarRocks for computation

4.2 Layer Responsibilities & Selection Specs

Partition Pruning (First Priority, Foundation): Design by time ( dt / hour). Rule: Bitmap/BloomFilter must pair with partition ; Range-Bitmap syntax doesn't enforce, but production massive data recommends mandatory.

BloomFilter Index (Second Priority, High-Cardinality Fields): Target fields: order_id, user_id etc. Role: file-level fast empty check, avoid full file scans.

Bitmap/Range-Bitmap Index (Third Priority, Filtering Core): Bitmap: low cardinality, equality/IN, strong partition dependency. Range-Bitmap: medium cardinality, range/mixed queries, partition optional.

4.3 Complete Production Table Creation + Index Config Template

-- Paimon Partitioned Table (Append-Only General)
CREATE TABLE paimon_order (
  order_id STRING,
  user_id BIGINT,
  status INT,
  channel INT,
  score INT,
  dt STRING
) PARTITIONED BY (dt)
WITH (
  'bucket' = '16',
  'file.format' = 'parquet',
  -- Layer 3: Bitmap low-cardinality equality filter (4 indexes, within optimal range)
  'file-index.bitmap.columns' = 'status,channel',
  'file-index.bitmap.status.index-block-size' = '16384',
  -- Layer 2: BloomFilter high-cardinality file pruning
  'file-index.bloom-filter.columns' = 'order_id,user_id',
  'file-index.bloom-filter.order_id.fpp' = '0.03',
  -- Medium cardinality range query: Range-Bitmap (partition optional)
  'file-index.range-bitmap.columns' = 'score'
);

4.4 Two Optimal Query SQL Comparisons

1) Full Index Combo (With Partition, Production Recommended, Best Performance)

SELECT * FROM paimon_order
WHERE dt = '2026-05-30'      -- Layer 1: Partition pruning (mandatory)
  AND order_id = 'O2026053001' -- Layer 2: BloomFilter file pruning
  AND status IN (1,2)          -- Layer 3: Bitmap row-level filter
  AND score > 60;              -- Range-Bitmap range filter

2) Pure Range-Bitmap (No Partition, Official Legal Syntax, Small Data Only)

-- Omit partition key dt, Range-Bitmap still works (Paimon official support)
SELECT * FROM paimon_order
WHERE score > 60 AND class_id IN (1,2);

4.5 Production Case (Map Trajectory Table)

Architecture: dt+hour dual partition + device_id (Bloom) + scene/device_type (Bitmap) + speed (Range-Bitmap)

Results:

Partitioned query: scanned data reduced 95%, latency from 5s to 200ms.

Non-partitioned full-table range query: Range-Bitmap filters normally, latency ~1.2s (acceptable for small business).

Production Red Line: For 10B/10M+ row tables, even though Range-Bitmap supports no-partition queries, must retain partition pruning to avoid full-table scan causing resource exhaustion.

Special Constraints: Bitmap Index Risks & Optimization in Primary Key Tables (Internals + Production Specs)

Paimon primary key tables use LSM-Tree architecture . Combined with File Index "data file + index 1:1 binding", updates/deletes trigger index rebuilds — key production constraint.

5.1 Why Primary Key Table Updates Trigger Index Rebuild

Updates/deletes do not modify data files in-place ; instead write new files + mark old data invalid via Deletion Vectors.

Background compaction merges old/new files, generating brand-new data files.

Due to 1:1 index-data binding, new data files must fully rebuild Bitmap/Range-Bitmap indexes ; old index files cleaned with old data files.

5.2 Core FAQ (High-Frequency Production Questions)

Data Loss? Official: No data loss . Paimon guarantees via MVCC, transaction atomicity, file checksums; data+index write is atomic, failure auto-rollbacks.

Query Impact? No query blocking, only short-term performance fluctuation : compaction raises CPU/IO, query latency +10%~50%; temporary index files cause storage spike (≤20%), auto-recycled.

Range-Bitmap on Primary Key Table Without Partition? Functionally supports no-partition queries, but continuous updates + file fragmentation amplify performance volatility; primary key tables strongly recommended to pair with partition .

5.3 Primary Key Table Bitmap Mandatory Specs (Official + Industry Consensus)

Count Red Line: Bitmap/Range-Bitmap total ≤ 3 , strictly forbidden >5.

Field Selection: Only create for low-update-frequency low-cardinality fields ; high-frequency update fields banned from bitmap indexes.

Must Enable: deletion-vectors.enabled=true to reduce file rewrite frequency, lower index rebuild pressure.

Auxiliary Optimization: Enable async compaction to avoid merge tasks contending query resources.

Partition Requirement: Primary key tables must configure partition in production regardless of index type.

5.4 Primary Key Table Standard Config Template

CREATE TABLE paimon_pk_order (
  order_id STRING PRIMARY KEY NOT ENFORCED,
  status INT,
  channel INT,
  score INT,
  user_id BIGINT,
  dt STRING
) PARTITIONED BY (dt)
WITH (
  'bucket' = '16',
  -- Primary key table mandatory: reduce file rewrites & index rebuilds
  'deletion-vectors.enabled' = 'true',
  -- Async compaction, isolate query & merge pressure
  'compaction.async.enabled' = 'true',
  -- Bitmap index total controlled within 3
  'file-index.bitmap.columns' = 'status,channel',
  'file-index.range-bitmap.columns' = 'score',
  -- High-cardinality high-frequency fields use BloomFilter
  'file-index.bloom-filter.columns' = 'user_id'
);

Index Validity Verification Methods (Production Troubleshooting Essential)

6.1 StarRocks Side: EXPLAIN Execution Plan

Run EXPLAIN to analyze query; check keywords to confirm index hits and observe partition usage:

-- Partitioned query
EXPLAIN SELECT * FROM paimon_order WHERE dt='2026-05-30' AND status=1 AND score>60;

-- Non-partitioned query (Range-Bitmap only)
EXPLAIN SELECT * FROM paimon_order WHERE score>60;

Summary & Enterprise-Grade Landing SOP

Core Essence: Paimon File Index 1:1 bound to data files; Bitmap is low-cardinality equality query dedicated row-level index; file structure/storage rules defined by official binary format.

Partition Rules (New Emphasis): Bitmap / BloomFilter strongly depend on partition pruning , no partition = terrible performance. Range-Bitmap Paimon officially states partition key not required , functionally can independently complete full-table filtering, but massive data/primary key scenarios still recommend pairing with partition.

Field Identification: SQL uses = / IN primarily, field low cardinality (≤10000 unique values), low update frequency → create Bitmap.

Count Red Lines: Append-Only table bitmap indexes 3~5 optimal, ≤8 ceiling; Primary key table bitmap indexes ≤3, strictly no exceed.

Index Replacement: Medium cardinality → Range-Bitmap, high cardinality → BloomFilter, extreme cardinality → B-Tree global index.

Optimal Architecture:

General: Partition Pruning + BloomFilter + Bitmap three-layer filtering.

Range query: Prefer Range-Bitmap, small data may omit partition, large data must stack partition.

Primary Key Table Special Constraints: Updates trigger index rebuild, no data loss, only minor query perf impact; must strictly control index count, enable Deletion Vectors and partition.