Apache Auron Graduates to TLP: 2x Spark Speedup via Rust-Native Vectorized Engine

Apache Auron Graduates to Top-Level Project

On October 14, the Apache Software Foundation announced that Apache Auron has graduated from the Apache Incubator to become a Top-Level Project (TLP). The graduation proposal passed the Incubator PMC (IPMC) vote in September 2026 and received Apache Board approval. Auron entered the incubator in August 2025 after being donated by Kuaishou, where it originated from large-scale data computing practice. The incubation-to-graduation journey took only 13 months.

Project Origin and Technical Architecture

Auron is a native vectorized execution accelerator for big data engines, currently focused on accelerating Apache Spark query processing. It combines Apache Arrow-DataFusion's native compute capabilities with Spark's distributed framework, translating Spark's optimized physical plans into equivalent native execution plans run by a Rust-based Native Engine.

The core architecture consists of four components:

Spark Extension : Mounts the accelerator into Spark's execution flow; users install Auron as a client extension and most SQL runs unchanged.

Flink Extension : Mounts the accelerator into Flink's execution flow.

Auron Core : Uses JNI to wrap the Native Engine, providing Java interfaces for engine integration.

Native Engine (Rust) : Built on Apache Arrow-DataFusion, includes execution plan specification, JNI gateway, custom operators/expressions/functions, memory management, and a fallback framework.

Key Technical Capabilities

Native execution with predictable performance : Rust core avoids JVM overhead, delivering stable, deterministic execution.

Vectorized computation : Leverages Apache Arrow columnar format and SIMD instructions for batch processing, maximizing throughput.

Comprehensive fallback mechanism : Unimplemented operators/expressions automatically fall back to Spark native execution; supports UDF/UDTF/UDAF, ensuring "if Spark runs it, Auron runs it."

Pluggable, multi-engine architecture : Deep Spark integration with extensibility via DataFusion to support Flink and other engines.

Rich ecosystem integration : Compatible with Apache Paimon, Iceberg, Hudi; shuffle services Apache Celeborn and Uniffle; storage systems HDFS, S3.

Production-hardened features : Multi-level memory management, compact shuffle format, adaptive execution strategies forged in large-scale production.

Benchmark Results and Production Validation

Auron has passed all TPC-DS and TPC-H benchmark tests. Compared to Apache Spark 3.5, Auron achieves over 2x speedup on TPC-DS while saving approximately 50% of compute resources . In production, it has been validated at exabyte (EB) scale , covering complex scenarios like JSON parsing and UDF/UDTF, and has resolved multiple stability and data consistency issues.

Deployed in production at dozens of companies including Kuaishou, DiDi, 58.com, Ctrip, OPPO, LinkedIn, and Autohome, delivering significant cost-efficiency gains.

Community Growth and Governance

Three releases during incubation.

Master branch: 179 commits merged in the last 6 months.

Contributors from 20+ different companies.

18 Committers at graduation, all converted to PMC members per graduation resolution.

22-person PMC (18 committers + 4 mentors), forming a diverse, cross-organizational governance body.

Future Roadmap

The project outlines four focus areas post-graduation:

Expand application ecosystem : Enhance Flink stream/batch support, adapt to new storage formats like Lance/Vortex.

Heterogeneous hardware support : Extend from CPU vectorization to GPU and FPGA, further boosting operator performance.

Continuous production problem-solving : Increase operator/function coverage, address performance, stability, and data consistency issues.

Grow community and ecosystem : Welcome more committers/contributors, establish stable release cadence per Apache TLP standards.

Notably, the team is building Auron on GPU based on cuDF to extend vectorized execution from CPU to GPU, addressing AI-era high-performance data compute demands.

Industry Endorsements

"Auron's incubation was special — the initial team already had deep Apache community experience. Graduation is a natural outcome, but it's just the beginning. We expect Auron to keep driving innovation in big data compute." — Guo Qiang , ASF Member, Apache Auron Incubation Champion

"Good version compatibility and usability make Native Execution benefits easy to land in production. At LinkedIn we've seen performance and resource efficiency gains. Excited about cross-engine exploration including Flink." — Qin Jiangjie , ASF Member, Incubation Mentor, LinkedIn Principal Staff Software Engineer

"25,000+ tasks deployed, 65% acceleration coverage, average 45%+ speedup. Auron brings higher compute efficiency and resource utilization with low adoption barrier." — Yu Yi , 58.com Data Architecture Department Head

Getting Involved

Project website: https://auron.apache.org/ GitHub repository: https://github.com/apache/auron Issues/PRs: https://github.com/apache/auron/issues Developer mailing list: [email protected] (subscribe via [email protected])

References

[1] Apache announcement:

https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-new-top-level-projects-7

[2] IPMC vote thread:

https://lists.apache.org/thread/fd8nhg1t8mo6rq8hqojjknob4rfsrszy