Presto and Alluxio Integration for Iceberg: Architecture, Best Practices, and Future Work

The article introduces the collaboration between the Presto SQL engine and the Alluxio storage system, focusing on their joint use with the Iceberg table format in modern data lake environments.

It first provides an overview of Presto, highlighting its ability to run interactive sub‑second queries on petabyte‑scale data, its two main open‑source communities (prestodb and trinodb), and typical use cases such as BI dashboards, A/B testing, and limited ETL workloads.

The core Presto architecture is described, where a coordinator parses SQL into execution plans and distributes work to multiple workers that read data from various connectors (Hive, Iceberg, Hudi, Kafka, Druid, MySQL, etc.), enabling data federation.

When combined with Alluxio, Presto can access data through a unified namespace, allowing workers and Alluxio workers to be co‑located on the same host for data locality, or deployed in a disaggregated fashion to avoid memory pressure at large scale.

The article then discusses Alluxio’s integration with Iceberg, noting challenges with schema evolution and metadata consistency, and presents two architectural options: (1) writing all data through Alluxio to guarantee consistency, and (2) reading and writing via Alluxio with automatic metadata synchronization, while acknowledging current limitations.

Best‑practice recommendations include using Iceberg’s native catalog (avoiding Hive‑Metastore locks), leveraging local SSD caches for Parquet files (via Presto’s RaptorX project), encrypting Parquet footers for data security, and enabling predicate push‑down to reduce scanned data volumes.

Future work aims to reduce CPU bottlenecks caused by Parquet parsing by separating hot and cold data, employing off‑heap Arrow or FlatBuffer formats for caching, and pushing more operators down to native storage layers to improve performance and lower GC pressure.

The article concludes with a brief thank‑you from the presenter.