Tencent Alluxio (DOP) Deployment and Optimization in Financial Data Analytics

Business Background

In Tencent's financial scenario, data analysis is accessed through two main portals: an SQL‑based platform (idex) and a visual BI tool (similar to Tableau) called "全民 BI". To support growing analytical demands, the data team upgraded the architecture by adding Presto and Tencent Alluxio (DOP) to enable free exploration of massive financial datasets.

Challenges

The OLAP workload faced two key issues: (1) rapid data growth combined with the need for high‑performance, low‑cost exploration, where SSDs alone were too expensive for large central storage; (2) mixed workloads (ETL and OLAP) causing IO bottlenecks on HDD‑based central storage, making random‑access performance critical.

Typical solutions copy hot data to dedicated storage to isolate IO, but this introduces data‑boundary and authentication‑consistency problems, especially in a regulated financial environment.

Alluxio Solution

Alluxio serves as a transparent caching layer with a full view of the underlying file system, preserving permissions and authentication. It offers two deployment modes: (1) cache‑accelerated mode tightly coupled with the compute engine for better IO locality; (2) IO‑isolation mode with separate Alluxio workers, allowing independent scaling.

To integrate Alluxio without modifying Hive metadata, a whitelist module redirects specific tables to Alluxio while leaving others untouched. An adaptive client layer also enables Presto, Spark, Flink, etc., to use whitelist or time‑range restrictions.

Additional challenges included preventing large‑range queries from evicting cached data, handling heterogeneous worker capacities, and avoiding excessive block replication. Solutions involved a time‑range based whitelist, limiting async cache threads, and a capacity‑based random block selection policy (CapacityBaseRandomPolicy) that assigns workers probabilities proportional to their storage capacity.

Performance Results

Real‑world query replay tests showed 68% speedup during low‑load weekends (500 queries) and around 300% speedup during busy weekday mornings (300 queries), demonstrating both SSD acceleration and IO isolation benefits.

Optimization and Tuning

The team adopted Tencent's KonaJDK with an optimized G1GC, reducing GC pauses. Using Kona‑profiler, they identified OOM caused by heavy finalizer usage in RocksDB's ReadOptions and upgraded RocksDB to a newer version. Moving block location metadata to memory cut query latency from 120 s to 28 s.

Other fixes included shortening the client‑worker graceful‑shutdown timeout, limiting the amount of metadata displayed on the Alluxio master UI to avoid page‑load stalls, and configuring dynamic thresholds for UI data exposure.

Summary and Outlook

Alluxio (DOP) provides a hierarchical BlockStore with a caching front‑end and persistent back‑end, supporting selective persistence of blockLocation data. Successful deployment in financial analytics required close collaboration between business, Alluxio developers, and JVM experts, and the experience paves the way for further scaling and feature enhancements.