Spark 4.0: New Features, Performance Gains, and Why It Still Leads Big Data

Following a recent essay on the future of Apache Hive, this article turns its focus to Apache Spark.

On May 23, 2025, Spark 4.0 was released, yet it generated little buzz because the Chinese community is currently dominated by Flink and a wave of lake‑house frameworks, while AI large‑model hype further overshadows Spark.

Nevertheless, Spark remains a cornerstone of data engineering: its GitHub repository has over 41.4K stars—the highest among data‑development frameworks—and maintains roughly 200 pull requests per year. Many large enterprises consider Spark the most competitive single compute framework, and interviewers often test candidates on Spark expertise.

The official release notes describe Spark 4.0 as a "significant milestone" and span 45 pages. Highlights from the release include:

- A new lightweight Python client (pyspark-client) at just 1.5 MB.
- An additional release tarball with Spark Connect enabled by default.
- Full API compatibility for the Java client.
- A new spark.api.mode configuration to easily turn on/off Spark Connect for your applications.
- Greatly expanded API coverage.
- ML on Spark Connect.
- A new client implementation for Swift.

Key improvements worth noting are:

PySpark Enhancements : native plotting, a Python data source API, and support for polymorphic user‑defined table functions (UDTF) dramatically boost Python developer productivity.

Spark Connect GA : decouples client and driver, supports lightweight clients in Go, Python, and other languages, and allows remote cluster debugging directly from a text editor, lowering the development barrier.

SQL and Query Optimization : adds SQL script support, ANSI‑SQL compatibility mode, dynamic partition pruning, and Adaptive Query Execution (AQE) to improve Spark SQL performance and compatibility.

Performance continues to be a core focus, with vectorized execution receiving major attention. Internally, Kuaishou’s open‑source vector engine Blaze can deliver up to a 30% compute boost.

Further optimizations include cross‑language execution via LLVM or JIT compilation and deeper integration with the Rust ecosystem to reduce execution overhead.

In the AI arena, Spark is expanding support for seamless integration with TensorFlow and PyTorch and exploring distributed training frameworks. The 2025 Spark roadmap mentions AI‑driven query optimization, such as automatic tuning and execution‑plan generation, as well as integration with large language models.

While Hive has adapted to stay relevant, Spark continues to lead the data‑development field, embracing challenges like vectorized execution, AI fusion, cloud‑native support, and cross‑framework collaboration, positioning itself for future trends.