How Zhihu Boosted Spark Job Performance with Gluten: A Practical Case Study

Background

Databricks published the Photon paper at ACM SIGMOD 2022, showing a 4× speedup on TPC‑H 3000SF and an open‑source counterpart, Gluten, achieving 2.7× faster than Spark 3.2 on TPC‑H 2000SF. Zhihu runs more than 10,000 Spark SQL jobs daily and has long sought to improve job performance and resource consumption.

Initial experiments in 2023 revealed serious shuffle stability problems, so the team focused on shuffle reliability and Spark version upgrades. After a pause, a manual dual‑run validation in early 2025 showed promising gains, but frequent compatibility issues led to another pause. Inspired by talks from ByteDance and Ant Group in late 2025, the migration effort was restarted in Q1 2026.

Gluten Overview

Gluten is a middle‑layer that offloads JVM‑based Spark computation to native engines such as ClickHouse and Velox. Its primary goal is to combine the high performance of native execution with the scalability and ecosystem of Spark.

Reuse Spark’s distributed control flow.

Translate Spark’s physical plan to Substrait, then to the native engine’s plan.

Execute performance‑critical paths in the native engine.

Velox Overview

Velox is an open‑source C++ execution engine that provides vectorized scalar, aggregate and window functions, relational operators, extensible I/O connectors, a unified type system, columnar memory layout compatible with Arrow, expression evaluation, network serializers, and resource‑management primitives.

Why Gluten Is Faster

The core computation is delegated to Velox, a pure C++ implementation, which avoids the overhead of JVM interpretation and JIT warm‑up. All data are stored column‑wise, enabling vectorized processing that fully utilizes CPU SIMD instructions (e.g., AVX‑512 can process 16 integers per instruction). Columnar storage also improves cache locality and reduces function‑call overhead in tight loops.

Job Migration Process

During the feasibility phase, the team used manual dual‑run testing on the TPC‑DS benchmark. Later, an automated migration pipeline was built (see image). Migration focused on the top‑resource‑consuming jobs; on average, migrated jobs saved about 40% of resources, but many compatibility and consistency problems were encountered, causing a temporary halt.

Data Consistency Issues

Two main categories were identified:

Jobs containing unstable computation logic (e.g., RowNumber, CollectSet, Rand, First). The team added custom Spark optimizer rules to replace these functions with stable equivalents (e.g., RowNumber → Rank, CollectSet → SortArray, Rand → hash of other columns).

Logical mismatches between Spark and Gluten/Velox. Representative cases include:

GCC vectorization bug : Over‑aggressive GCC inlining removed a necessary memory‑to‑register copy, causing incorrect results. The issue was isolated by narrowing GCC optimization flags and fixing the affected fixedWidthScan branch.

GetJsonObject inconsistency : Velox’s Simd library did not support wildcard paths, and its string rendering added extra spaces. The team upgraded Simd, patched Velox, and contributed a bug report (SimdJSON issue #2684). Example: Spark returns [{"extended_type":"FeedDeliveryType","value":"Normal"}] while Velox returns [{"extended_type": "FeedDeliveryType","value": "Normal"}].

Parquet 1.8.1 string index filter : Older Parquet versions sort strings as signed bytes, while Velox assumes unsigned sorting, leading to wrong filter results. The fix skips index filtering for those versions.

Decimal storage mismatch : Spark writes decimals as FIXED_LEN_BYTE_ARRAY (legacy format), while Velox chooses the smallest possible storage. This caused compatibility errors with Hive 2.1.1. A patch aligned the storage format.

NativeUnion bug : Field type names were mistakenly used as field names during plan conversion, producing identical column values. The issue was fixed in a Velox PR.

Performance Issues

A dual‑run test of the top 200 resource‑heavy jobs showed that roughly 40% of jobs had no noticeable benefit or even regressed (≈20% negative, 20% <10% gain). Most of these were IO‑heavy, spill‑heavy, or contained many Union operators. The team identified two main causes:

Async prefetch contention : Gluten+Velox uses a shared thread pool for prefetching IO. On clusters with slow HDFS nodes, a single task could occupy all prefetch threads, throttling other tasks. Setting spark.gluten.sql.columnar.backend.velox.IOThreads=0 disables prefetch and restored IO performance.

Spill handling : Velox’s spill process involves multiple column‑row conversions (column → row for aggregation state, row → column for spill, column → row for merge). Enabling prefix‑sort (

spark.gluten.sql.columnar.backend.velox.spillPrefixsortEnabled=true

) reduced spill overhead for many jobs.

Stability Issues

Several crash scenarios were observed and fixed:

GCC vectorization causing CoreDump : LZO‑compressed Parquet files triggered a crash due to over‑wide vector instructions writing beyond buffer bounds. Disabling vectorization for lzoDecompress resolved the issue (Velox PR #13453).

Libhdfs JNI CoreDump : When a C/C++ thread created by the JVM terminated, the JNI environment was already destroyed, leading to invalid function‑pointer calls. The problem aligns with Hadoop JIRA HDFS‑16021; a community patch was applied.

Decimal handling incompatibility : Spark’s legacy Parquet format stored decimals as FIXED_LEN_BYTE_ARRAY, while Velox used size‑optimized storage, causing Hive/Flint read failures. A compatibility patch was merged (Velox PR #16941).

CaseWhen with get_struct_field : Velox reused a single result vector for all branches, but get_struct_field created a new vector, breaking the contract and producing wrong results. The function was updated to reuse the passed‑in vector when possible (Velox PR #17381).

Parquet Array null‑buffer bug : When reading multiple RowGroups with differing row counts, Velox reused a nulls buffer of size 4096 for a batch of 4006 rows, leaving the buffer state inconsistent and causing execution failures. The bug was fixed (Velox PR #17496).

Benefits and Future Plans

By focusing on the top‑resource jobs, Zhihu successfully migrated 2,446 out of 3,200 candidates (76% success). The migration saved approximately 130,000 core‑hours (44.6% reduction) and 280,000 GB of memory (57% reduction). The resource‑coverage now reaches about 80% of total Spark SQL consumption.

Future work includes:

Addressing the remaining failed jobs, which are mainly IO‑heavy, spill‑heavy, or text‑format jobs.

Exploring data‑synchronization pipelines and Spark‑Jar job migration.

Evaluating Photon’s Parquet write performance (expected 2× speedup) and extending it to Zhihu’s workloads.

Investigating Gluten migration for Flink jobs, leveraging community reports of a 3× throughput increase on the Nexmark benchmark.