How lance‑spark Implements Blob V2 Support: A Deep Dive

1. Enabling Blob V2 in lance‑spark

lance‑spark supports Lance's Blob V2 when a column is declared with blob encoding and the table property file_format_version is set to at least 2.2. An example table definition is:

CREATE TABLE IF NOT EXISTS lance.dwd_insys_test.lance_v8_target_example (
    id BIGINT NOT NULL COMMENT '行id',
    description STRING COMMENT '描述',
    image_embedding ARRAY<FLOAT> COMMENT '图片embedding向量',
    raw_data BINARY COMMENT '二进制blob数据'
) USING lance COMMENT 'lance-spark下Blobv2目标表'
TBLPROPERTIES (
    'image_embedding.arrow.fixed-size-list.size' = '128',
    'raw_data.lance.encoding' = 'blob',
    'file_format_version' = '2.2'
);

When reading, Spark returns a STRUCT descriptor instead of the raw BINARY:

STRUCT<kind: SMALLINT, position: BIGINT, size: BIGINT, blob_id: BIGINT, blob_uri: STRING>

Write interface still accepts Spark BINARY.

Actual binary bytes are stored in Lance data files, shared .blob files, dedicated .blob files, or external URIs depending on size.

The main table column only keeps a small locator descriptor.

Normal Spark scans return only the descriptor; the full bytes are fetched only via takeBlobs.

2. Why Spark Can Still Write Plain BINARY

During table creation, SchemaConverter.processSchemaWithProperties scans each column for the property raw_data.lance.encoding = blob. If present, the column must be of Spark BinaryType and metadata ARROW:extension:name = lance.blob.v2 (or lance-encoding:blob for older versions) is added.

When converting to an Arrow schema, LanceArrowUtils.scala checks this metadata. If it equals lance.blob.v2, the generated Arrow field is a struct with data, uri, position, and size instead of a plain LargeBinary.

The writer BlobV2StructWriter then writes the byte array into the data sub‑field while leaving the other three sub‑fields null.

if (input.isNullAt(ordinal)) {
    valueVector.setNull(count);
} else {
    valueVector.setIndexDefined(count);
    dataWriter.write(input, ordinal);
}

3. Size‑Based Placement of Binary Content

In the Rust side ( dataset/blob.rs), preprocess_blob_array decides where to store the bytes based on three thresholds:

const INLINE_MAX: usize = 64 * 1024;          // 64 KiB
const DEDICATED_THRESHOLD: usize = 4 * 1024 * 1024; // 4 MiB
const PACK_FILE_MAX_SIZE: usize = 1024 * 1024 * 1024; // 1 GiB

The resulting storage kind is:

Inline (0) : size ≤ 64 KiB → stored in the out‑of‑line buffer of the main data file.

Packed (1) : 64 KiB < size ≤ 4 MiB → stored in a shared .blob file.

Dedicated (2) : size > 4 MiB → stored in an exclusive .blob file.

External (3) : data referenced by a URI → stored outside the dataset.

For example, a 1 MiB raw_data value follows the Packed branch, causing the bytes to be written to a shared .blob file and only the locator fields (kind, blob_id, size, position) are kept in the main column.

4. Converting the Write Structure to a Persistent Descriptor

After preprocessing, BlobV2StructuralEncoder creates the final descriptor defined in datatypes.rs:

Struct<kind: UInt8, position: UInt64, size: UInt64, blob_id: UInt32, blob_uri: Utf8>

Each storage kind fills the fields as follows:

Inline : kind = 0, position = offset in OOL buffer, size = length, blob_id = 0, blob_uri = "".

Packed : kind = 1, position = offset inside shared .blob, size = length, blob_id = shared file id, blob_uri = "".

Dedicated : kind = 2, position = 0, size = file length, blob_id = dedicated file id, blob_uri = "".

External : kind = 3, position = start offset in external object, size = read length, blob_id = base‑path id (0 for absolute URI), blob_uri = external URI.

5. Why Spark Queries Return a STRUCT Instead of BINARY

lance‑spark deliberately maintains two schemas:

Write schema: BinaryType + lance.blob.v2 metadata.

Read schema: Blob descriptor StructType. LanceDataset.schema() calls BlobUtils.applyBlobV2DescriptorSchema(sparkSchema), which replaces the original raw_data: BinaryType with the descriptor struct. The scan builder performs the same conversion, so Spark’s analyzer sees the descriptor from the start and query results are of the form:

SELECT raw_data.kind, raw_data.position, raw_data.size, raw_data.blob_id, raw_data.blob_uri FROM target;

Ordinary scans therefore never materialise the full binary payload, avoiding costly large‑object reads.

6. Mapping Between Lance Arrow Types and Spark Types

Lance’s native Arrow descriptor uses unsigned integers ( UInt8, UInt32, UInt64). Because Spark lacks unsigned types, LanceArrowUtils.scala maps them as:

Arrow UInt8  → Spark ShortType
Arrow UInt32 → Spark LongType
Arrow UInt64 → Spark LongType

Consequently the Spark‑visible struct appears as:

STRUCT<kind: SMALLINT, position: BIGINT, size: BIGINT, blob_id: BIGINT, blob_uri: STRING>

7. Actual Byte Retrieval with takeBlobs

When the real blob content is needed, the user calls dataset.takeBlobs(rowAddresses, column). The Rust side collect_blob_entries_v2 dispatches based on kind:

Inline : read from the fragment’s data file using position..position+size.

Packed : locate the shared .blob file via blob_id and read the specified range.

Dedicated : open the dedicated .blob file identified by blob_id.

External : access the external object storage using blob_uri and read the range.

The additional _rowaddr field helps resolve which fragment a descriptor belongs to for Inline, Packed and Dedicated cases.

8. Special Representation of Null Blobs

Blob V2 descriptors are non‑nullable. A null blob is represented by a sentinel descriptor where all fields are zero (or empty string for blob_uri):

kind = 0, position = 0, size = 0, blob_id = 0, blob_uri = ""

Spark’s BlobUtils recognises this pattern to distinguish a true null from a valid but empty descriptor.