Airbnb OpenAir Conference: Open‑Source Tools Airpal, Aerosolve, and Airflow

On July 5, Airbnb held its first OpenAir technology conference, focusing on data‑driven practices in its development process, and announced three open‑source big‑data tools.

Airpal

The first tool, Airpal, is Airbnb’s most popular internal data‑analysis platform and has gathered over 900 stars on GitHub. Airpal is built on Facebook’s Presto and provides a visual, distributed SQL query engine. Airbnb now stores roughly 1.5 PB of data; traditionally this data was queried with Hive, which has several drawbacks.

First, for small queries the MapReduce overhead of Hive is excessive—for example, a simple SELECT * FROM table LIMIT 10 triggers a full MapReduce job that can take half a minute just in the map phase. Presto, used by Airpal, avoids this latency and also offers a preview of table data.

Second, Hive’s command‑line interface is unfriendly to non‑technical users. Airpal’s graphical interface requires only SQL knowledge and can export results directly to CSV, making it convenient for analysts in finance or other departments. In practice, Airbnb data scientists still prefer Hive’s CLI, while engineers and product managers performing simple queries rely on Airpal.

Airpal also integrates with the company’s LDAP system, allowing employees to log in with their corporate credentials and automatically receive appropriate data‑access permissions.

Aerosolve

Aerosolve is the machine‑learning engine that powers Airbnb’s pricing recommendation system. Unlike traditional black‑box ML engines, Aerosolve is designed to be interpretable, enabling users to see which features most influence the model’s output.

For instance, the system can explain that a listing’s price is affected by the number of reviews and the count of three‑star reviews, showing that beyond a single review, additional reviews have diminishing impact, and an excess of three‑star reviews may even be detrimental.

Airflow

The foundation of big‑data work remains the data pipeline. Airflow is Airbnb’s internal tool for launching, ordering, and monitoring these pipelines.