MaxCompute: Intelligent Data Warehouse Platform for the Data+AI Era

This article, based on a meetup talk about building intelligent data warehouse platforms in the Data+AI era, introduces Alibaba Cloud's MaxCompute, its evolution, and core capabilities.

MaxCompute is a cloud‑native, serverless big‑data compute service offering elastic resource scheduling, multi‑tenant security, and support for both offline and near‑real‑time analytics, as well as AI‑driven workloads.

The platform integrates a self‑developed SQL engine, open‑source engines such as Apache Spark, and a distributed Python engine, and adopts a storage‑compute separation architecture built on Alibaba Cloud’s Feitian storage and OpenLake lakehouse.

Key challenges for data warehouses in the Data+AI age are identified: the need for generative AI, handling heterogeneous data for model pre‑training, AI‑enhanced warehouse features, and rapid development‑test‑deploy cycles.

To address these, MaxCompute provides OpenLake data management, a Python‑centric distributed framework called MaxFrame, an interactive notebook‑like development environment, and a custom image management platform for UDFs.

MaxFrame unifies Python APIs, supports Pandas‑compatible operations, and enables seamless use of Object Table, which exposes OSS object metadata as SQL tables and allows document‑function‑based processing of unstructured data.

Object Table empowers AI scenarios such as image analysis by letting users read files directly from the data lake without custom scripts, and supports generation of structured data from unstructured sources.

Near‑real‑time capabilities include MCQA 2.0 interactive query engine, incremental MV pipelines, DeltaTable with minute‑level checkpoints, and automatic storage optimizations.

AI Function offers GenAI services backed by Alibaba’s Feitian large‑model, enabling simple API‑driven image content analysis within MaxCompute workflows.

The intelligent data‑warehouse features—smart diagnostics, materialized view automation, query optimization, and data layout recommendations—demonstrate how AI can further enhance warehouse performance.