What Spark 4.0 Brings: VARIANT Type, Native SQL UDFs, and Serverless Enhancements

Apache Spark 4.0 Overview

Apache Spark 4.0 is the most significant release since the project’s inception. It introduces a binary VARIANT data type for semi‑structured data, native SQL user‑defined functions (UDFs), table‑returning functions, a pure‑Python DataSource API, a new pipeline operator ( |>), enhanced Structured Streaming state management, and a set of serverless infrastructure upgrades that together deliver roughly 30% faster query execution.

1. VARIANT Data Type – Efficient Semi‑Structured Storage

In earlier Spark versions JSON payloads were stored as STRING, requiring full parsing for each field and preventing column pruning or predicate push‑down. The VARIANT type stores JSON in a binary, indexed format, enabling O(1) path lookups, optimizer awareness, and dynamic schema evolution.

-- Traditional STRING table
CREATE TABLE user_events (
    event_id BIGINT,
    raw_payload STRING -- JSON as plain text
);
SELECT get_json_object(raw_payload, '$.user_id')   AS user_id,
       get_json_object(raw_payload, '$.event_type') AS event_type
FROM user_events
WHERE get_json_object(raw_payload, '$.event_type') = 'page_view';

-- VARIANT table
CREATE TABLE user_events (
    event_id BIGINT,
    payload VARIANT -- binary‑encoded, automatically indexed
);
INSERT INTO user_events SELECT 1, parse_json('{"user_id":"U12345","event_type":"purchase"}');
SELECT payload:user_id::STRING   AS user_id,
       payload:event_type::STRING AS event_type
FROM user_events
WHERE payload:event_type::STRING = 'purchase';

Key advantages of VARIANT over STRING:

Storage format: binary with automatic indexing vs plain JSON text.

Query performance: O(1) path location vs O(N) repeated parsing.

Optimizer support: path expressions participate in predicate push‑down.

Schema flexibility: dynamic adaptation to evolving structures.

Syntax simplicity: intuitive payload:field::TYPE notation.

2. Native SQL UDFs – Removing Python/Java Overhead

Spark 3.x required Python or Java UDFs, which introduced cross‑process serialization costs and prevented the optimizer from analysing the function body. Spark 4.0 allows pure‑SQL UDF definitions that are inlined and fully optimized.

-- Python UDF (Spark 3.x)
from pyspark.sql.functions import udf
from pyspark.sql.types import DoubleType
@udf(returnType=DoubleType())
def calculate_discount(price, member_level):
    rates = {1: 0.95, 2: 0.90, 3: 0.85}
    return price * rates.get(member_level, 1.0)

-- SQL UDF (Spark 4.0)
CREATE FUNCTION calculate_discount(price DECIMAL(10,2), level INT)
RETURNS DECIMAL(10,2)
RETURN CASE level
    WHEN 1 THEN price * 0.95
    WHEN 2 THEN price * 0.90
    WHEN 3 THEN price * 0.85
    ELSE price END;

SELECT * FROM orders WHERE calculate_discount(price, member_level) > 1000;

SQL UDFs can be composed; the optimizer expands them, enabling constant folding, predicate push‑down, and eliminating the Python‑JVM bridge.

3. Table‑Returning Functions

Spark 4.0 supports functions that emit a table, allowing multi‑row results directly in SQL.

CREATE FUNCTION date_range(start_date DATE, end_date DATE)
RETURNS TABLE(dt DATE, day_of_week STRING)
RETURN SELECT day, date_format(day, 'EEEE')
FROM (SELECT sequence(start_date, end_date)) AS T(days)
LATERAL VIEW explode(days) AS day;

SELECT * FROM date_range('2025-01-01', '2025-01-31');

4. Python DataSource API – Pure‑Python Connectors

Developers can now implement custom data sources entirely in Python without any Java/Scala scaffolding. After registering the connector, it behaves like built‑in formats (Parquet, CSV).

# Register a Python data source implementation
spark.dataSource.register(OssJsonDataSource)

# Write data
df.write.format("oss_json").option("path", "data/events").mode("overwrite").save()

# Read data
spark.read.format("oss_json").option("path", "data/events").load().show()

5. Pipeline Syntax – Aligning Query Order with Data Flow

The new |> operator lets users write queries in the same top‑down order as the data processing pipeline, improving readability.

-- Traditional nested query
SELECT region, total FROM (
    SELECT region, SUM(amount) AS total FROM orders GROUP BY region
) WHERE total > 100000 ORDER BY total DESC;

-- Pipeline syntax
FROM orders
|> AGGREGATE SUM(amount) AS total GROUP BY region
|> WHERE total > 100000
|> ORDER BY total DESC;

6. Structured Streaming State Management v2

Arbitrary State API v2 – manage multiple state variables within a single operator.

State DataSource – direct read and debugging of streaming state.

Reduces development and operational complexity for stateful streams.

7. Infrastructure Upgrades (Serverless EMR)

Alibaba Cloud EMR Serverless Spark fully adapts Spark 4.0 and adds several runtime enhancements:

Paimon Variant integration : column‑store optimisation, predicate push‑down, and compile‑time type safety for JSON‑heavy workloads.

Fusion vectorised engine : up to 3× speed‑up over open‑source Spark on TPC‑DS benchmarks.

Native Python UDF execution : eliminates cross‑process overhead.

Zero‑code migration : automatic compatibility with JDK 8 jobs (Spark 4.0 runs on JDK 17+), and default ANSI‑SQL mode disabled to preserve Spark 3.x behaviour.

8. Performance Highlights

Benchmarking on TPC‑DS shows Spark 4.0 delivers roughly 30% lower query latency compared with Spark 3.x, thanks to optimizer refinements, the Fusion vectorised engine, and the VARIANT data type.

Overall, Spark 4.0 provides a high‑performance VARIANT type for semi‑structured data, native SQL UDFs that integrate with the optimizer, richer Python APIs, a more expressive pipeline syntax, advanced streaming state management, and a revamped serverless runtime that together enable faster, more maintainable data pipelines.