How Iceberg V3 and Spark 4.0 Redefine Lakehouse Development

In May 2026 Apache Iceberg 1.11.0 (the production‑ready V3) and Apache Spark 4.0 were released simultaneously. Iceberg V3 adds deletion vectors, the VARIANT data type, geospatial types and nanosecond timestamps, while Spark 4.0 on Amazon EMR natively supports VARIANT and introduces Spark Connect.

Iceberg V3 Core Changes

Deletion Vectors

In the V2 merge‑on‑read model, high‑frequency updates/deletes generated many positional delete files, causing Spark to open and compare a large number of small files, which severely degraded read performance. V3 replaces these files with a single Roaring bitmap stored in a Puffin file; each bit indicates whether a row is deleted, dramatically reducing read amplification.

This improvement is especially beneficial for real‑time upsert scenarios such as CDC or order‑status streams, where the same delete/update rate now incurs far less I/O pressure on the read side.

-- Create a V3 table with Deletion Vectors enabled
CREATE TABLE catalog.db.orders (
    order_id BIGINT,
    pay_time TIMESTAMP,
    payload  STRING
) USING iceberg
TBLPROPERTIES (
    'format-version' = '3',               -- key: declare V3
    'write.delete.mode' = 'merge-on-read',
    'write.update.mode' = 'merge-on-read',
    'write.delete.format' = 'puffin'   -- store bitmap in Puffin
);

VARIANT

V3 introduces a native binary‑encoded semi‑structured type called VARIANT, which can be stored directly instead of as a JSON string that must be parsed at query time.

Old approach (STRING + runtime parsing):

SELECT
    get_json_object(payload, '$.user.id')   AS user_id,
    get_json_object(payload, '$.event.type') AS event_type
FROM logs_raw
WHERE get_json_object(payload, '$.event.type') = 'click';

New approach (VARIANT + path access):

-- Table definition using VARIANT
CREATE TABLE logs_v (
    ts      TIMESTAMP,
    payload VARIANT   -- first‑class semi‑structured column
) USING iceberg
TBLPROPERTIES ('format-version' = '3');

-- Insert: parse_json encodes JSON into VARIANT once
INSERT INTO logs_v
SELECT ts, parse_json(raw_json) FROM logs_raw;

-- Query: variant_get extracts values by path with type hint
SELECT
    variant_get(payload, '$.user.id',   'long')   AS user_id,
    variant_get(payload, '$.event.type','string') AS event_type
FROM logs_v
WHERE variant_get(payload, '$.event.type','string') = 'click';

Spark 4.0

Beyond the data‑layer upgrade, Spark Connect changes the architecture. It decouples the client from the Spark driver by introducing a client‑server protocol that sends an unresolved logical plan (describing *what* to do) over gRPC/HTTP2 to a remote Spark server, which then parses, optimizes, executes, and returns results.

The client converts a DataFrame query into an unresolved logical plan.

The plan is encoded and transmitted to the Spark server.

The server performs analysis, optimization, execution, and sends back the result.

# Spark Connect: thin client connecting to a remote server
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .remote("sc://spark-server-host:15002") \
    .getOrCreate()

# The code looks identical to local usage, but runs remotely
spark.table("logs_v").limit(5).show()

Benefits include language‑agnostic clients (Go, Swift, Rust, etc.), isolation of client crashes from the server, lighter dependency footprints for IDEs, notebooks or micro‑services, and safer multi‑tenant operation. Spark 4.0 also makes ANSI SQL mode the default, causing overflow, division‑by‑zero and type‑conversion errors to raise explicit exceptions rather than silently returning null.

Conclusion

The Iceberg V3 + Spark 4.0 GA is more than a collection of new features; it resolves the hardest challenges of lakehouse workloads—semi‑structured data handling and high‑frequency upserts—by making VARIANT a first‑class type, using Deletion Vectors to control read amplification, and turning Spark into a protocol‑driven service via Spark Connect. The practical recommendation is to create new tables with V3 and VARIANT for such workloads.