Accelerating QueryEngine with Alluxio in Spark SQL: Architecture, Features, and Performance Evaluation

1 Introduction

With the rapid growth of Beike Zhaofang, the ad‑hoc query demand on the QueryEngine has increased dramatically, leading to larger data volumes per request and higher query pressure. Frequent data fetching from HDFS also strains network and disk I/O. To improve analysis speed and reduce I/O pressure, we introduced Alluxio as an intelligent caching middleware between QueryEngine and the big‑data cluster, temporarily storing data in memory or other fast media close to the compute service.

2 Overview

2.1 What is Alluxio

Alluxio (formerly Tachyon) is the world’s first memory‑centric virtual distributed storage system. It unifies data access, acting as a bridge between upper‑level compute frameworks and lower‑level storage systems. Applications only need to connect to Alluxio to access data residing in any underlying storage (e.g., HDFS, S3, Ceph). Its memory‑centric architecture provides access speeds several orders of magnitude faster than conventional solutions and is compatible with Hadoop ecosystems.

2.2 Alluxio Features

1) Hierarchical storage – manages memory, SSD, and HDD layers, automatically placing hot data on faster tiers.

2) Flexible file API – provides Java‑like InputStream/OutputStream and efficient memory‑mapped I/O; also offers Hadoop‑compatible filesystem interfaces.

3) Unified namespace – mounts multiple storage systems under a single path, preserving file names and directory hierarchy.

4) Data sharing – enables high‑speed sharing of intermediate results across different Spark jobs.

5) High read/write performance – delivers memory‑level latency for hot data.

6) Pluggable and fault‑tolerant – backs up memory data to underlying storage and provides generic interfaces for new storage systems.

7) Read‑write separation – suitable for architectures that separate reads and writes, allowing one‑time reads to be reused multiple times.

3 Practice

3.1 Why Use Alluxio

The QueryEngine currently uses three engines (MR, Tez, Spark SQL) and selects one based on cost. Spark SQL runs on a dedicated YARN cluster. Problems include large and growing data volumes, degraded performance, remote data loading for each query, and repeated accesses to largely static T+1 data, causing heavy I/O pressure on the storage cluster.

Alluxio’s features make it well‑suited to alleviate these issues by caching data close to compute.

3.2 Performance Evaluation

1) Test Environment

Hardware: 3 servers (48‑core CPU, 128 GB RAM, 5.4 TB HDD (11 disks), 3 TB SSD, 1 GbE).

Software: Hadoop‑2.7.3, Alluxio‑1.7.1, Spark‑2.3.0.

2) Test Data Size

Alluxio supports hierarchical storage (memory, SSD, HDD) with replacement policies (LRU, LFU). Three data sizes were selected to evaluate cold reads, cached reads, and replacement behavior.

3) Test Scenarios

Each scenario was executed twice to compare cold‑read efficiency and cache efficiency.

4) Test Results

Cold reads (First scenario) performed worse with Alluxio; data must be pre‑loaded for production.

For larger data sets (SQL2, SQL3), Alluxio’s eviction efficiency is low; careful data‑size planning is required to avoid frequent replacement and data inflation caused by passive caching.

When no eviction occurs, SSD‑backed Alluxio yields 2‑5× speedup.

In HDD scenarios, acceleration is modest but improves with more Alluxio workers.

4 Deployment

Single‑layer SSD storage gave the highest efficiency; to avoid replacement, production uses SSD‑only Alluxio while letting Linux handle RAM caching.

Given cost constraints and massive ad‑hoc query volumes (hundreds of TB), we cache only medium‑size, high‑frequency tables and partitions identified from the past 30 days of query logs.

For Spark SQL queries, we run Spark ThriftServer. We modified HiveMetaStore source code so that whitelisted tables resolve to Alluxio paths, while others read directly from HDFS.

We observed that the default AlluxioFileSystem loader is single‑threaded and slow; by parallelizing data loading with Spark jobs, cache population speed increased dramatically.

We also adjusted alluxio.user.file.metadata.sync.interval from 1 day to 1 hour to keep Alluxio metadata in sync with HDFS when upstream data is refreshed.

5 Other Alluxio Scenarios

1) Accelerate machine‑learning and deep‑learning workloads at Beike.

2) Enable data sharing across multiple frameworks.

3) Optimize Spark shuffle and JVM memory pressure.

4) Provide a unified view of heterogeneous file systems.

6 Conclusion

Alluxio, as an excellent in‑memory file system, offers many opportunities for performance improvement across various big‑data workloads.

7 References

http://www.alluxio.org/docs/1.7/en/Running-Alluxio-on-a-Cluster.html

http://www.alluxio.org/overview/performance-tuning

https://www.slidestalk.com/s/JD_Presto_On_Alluxio_2018

https://zhuanlan.zhihu.com/alluxio