Paimon File Indexes: Technical Reference for BloomFilter, Bitmap & Range Bitmap

1. File Index Overall Specification

1.1 Basic Definition

File-level indexes are enabled via the configuration parameter file-index.${index_type}.columns. When enabled, Paimon generates a separate index file for each data file.

Small index files: embedded directly in the Manifest (metadata manifest file).

Large index files: stored in the same directory as the corresponding data file.

Binding: one data file maps to one independent index file.

Composite capability: a single index file can carry multi-column, multi-type index data simultaneously.

1.2 Index File Common Format

All file-level index types share a unified file structure divided into HEAD and BODY regions. The head records column information, index offsets, and lengths; the body stores raw index byte streams for each type. All integer fields use big-endian (BIG_ENDIAN) byte order.

2. BloomFilter Index

2.1 Configuration Parameters

# Specify columns for BloomFilter index, comma-separated
file-index.bloom-filter.columns
# False positive probability per column
file-index.bloom-filter.<column>.fpp
# Estimated distinct values per data file for size optimization
file-index.bloom-filter.<column>.items

2.2 Binary Storage Structure

The BloomFilter index body consists of two parts (big-endian): numHashFunctions — 4-byte integer, number of hash functions. bloom filter bytes — Bloom filter bitmap byte stream.

2.3 Hash Rules

Based on 64-bit long integer hash.

String/binary types (varchar, binary, etc.): use XX Hash algorithm.

Numeric types: use dedicated numeric hash algorithm.

2.4 Core Capability

Used only for file-level existence checks , quickly filtering out data files that do not contain the target value. No row-level filtering capability; suitable for high-cardinality fields.

3. Bitmap Index

3.1 Basic Configuration Parameters

# Specify columns for Bitmap index, comma-separated
file-index.bitmap.columns
# Secondary index block size, default 16KB
file-index.bitmap.<column>.index-block-size

3.2 Mainstream Version: Bitmap V2 Storage Format

Current production uses V2, divided into three parts: HEAD , INDEX BLOCKS , BITMAP BLOCKS . All integers are big-endian.

Index Block Sub-structure

value x

— actual field value (variable-length bytes), unique identifier for the bitmap. offset — 4-byte integer. If negative, indicates the value exists in only a single row; actual position is the absolute value. length — 4-byte integer, current bitmap data byte length.

3.3 Legacy Version: Bitmap V1

All integers use big-endian; gradually replaced by V2.

3.4 Supported Data Types

Bitmap index supports only: TinyIntType, SmallIntType, IntType, BigIntType, DateType, TimeType, LocalZonedTimestampType, TimestampType, CharType, VarCharType, StringType, BooleanType.

3.5 Core Capability

Optimizes row-level exact equality and IN set queries ; bitmap intersection/union operations are efficient. Suitable for low-cardinality fields. Ineffective for range queries and fuzzy queries.

4. Range Bitmap Index

4.1 Feature Overview

Advantages

Index size smaller than traditional Bitmap index.

Supports both point queries and range queries , adapting to medium/high-cardinality scenarios.

Supports AND/OR composite predicate optimization (related columns must have Bitmap/Range-Bitmap indexes).

Supports TOPN/BOTTOMK sorted queries ( only for Append-Only Tables ).

Disadvantages

Pure point equality query execution efficiency is slightly lower than traditional Bitmap index.

4.2 Configuration Parameters

# Specify columns for Range-Bitmap index, comma-separated
file-index.range-bitmap.columns
# Data chunk size, default 16KB
file-index.range-bitmap.<column>.chunk-size

4.3 Core Usage Rules (Official Emphasis)

Range-Bitmap optimizes four query types: equality, range, AND/OR, TOPN . Bitmap and Range-Bitmap index results are merged and pushed down to data files for row-group/data-page filtering.

Official key note: When using Range-Bitmap for query optimization, partition keys (e.g., dt) are not mandatory conditions .

4.4 Optimization Scenarios & Standard SQL Examples

Prerequisite: table has class_id and score columns with Range-Bitmap indexes.

1) Equality Query Optimization

-- Single-value equality
SELECT * FROM TABLE WHERE dt = '20250801' AND score = 100;
-- IN set query
SELECT * FROM TABLE WHERE dt = '20250801' AND score IN (60, 80);

2) Range Query Optimization

SELECT * FROM TABLE WHERE dt = '20250801' AND score > 60;
SELECT * FROM TABLE WHERE dt = '20250801' AND score < 60;

3) AND/OR Composite Predicate Optimization

SELECT * FROM TABLE WHERE dt = '20250801' AND class_id = 1 AND score < 60;
SELECT * FROM TABLE WHERE dt = '20250801' AND class_id = 1 AND score < 60 OR score > 80;

4) TOPN Sorted Query Optimization

Constraint: TOPN optimization cannot be combined with other query predicates ; currently only Apache Spark engine supports it; for multi-field sorting, the first sort field must have a Range-Bitmap index .

-- Single-field ASC/DESC pagination
SELECT * FROM TABLE WHERE dt = '20250801' ORDER BY score ASC LIMIT 10;
SELECT * FROM TABLE WHERE dt = '20250801' ORDER BY score DESC LIMIT 10;
-- Multi-field sort (first sort field must have Range-Bitmap)
SELECT * FROM TABLE WHERE dt = '20250801' ORDER BY score ASC, col DESC LIMIT 10;
SELECT * FROM TABLE WHERE dt = '20250801' ORDER BY score DESC, col ASC LIMIT 10;

4.5 Range-Bitmap V1 Storage Format

Divided into three parts: HEAD , Dictionary Area , Bit-Slice Index Area .

Extends traditional Bitmap with full support for the following types: TinyIntType, SmallIntType, IntType, BigIntType, DateType, TimeType, LocalZonedTimestampType, TimestampType, CharType, VarCharType, StringType, BooleanType, DoubleType, FloatType.

5. Supplementary Technical Points Summary

Unified Byte Order Rule : All index file integer fields use big-endian (BIG_ENDIAN) .

Encoding Rule : Column names, index names, and other string fields uniformly use Java Modified UTF-8 encoding.

Scenario Selection Quick Reference :

Low cardinality + pure equality/IN queries → prefer Bitmap.

Medium/high cardinality + range/mixed queries → prefer Range-Bitmap.

High cardinality unique values + file-level filtering only → prefer BloomFilter.

Partition Constraint Differences :

Bitmap, BloomFilter: recommend adding partition pruning to reduce file scans and improve performance.

Range-Bitmap: officially states partition key is not required , can independently perform full-table filtering; however, massive data production scenarios still recommend combining with partitions for performance.

Table Type Constraint : Range-Bitmap's TOPN optimization only works for Append-Only Tables ; primary-key update tables do not support this capability.