How MaxCompute Evolved: 10 Years of Big Data Innovation at Alibaba

MaxCompute (ODPS) Overview

MaxCompute, Alibaba's EB‑scale computing platform, has become the core of the group's data middle platform and the foundation of Alibaba Cloud's big‑data services after ten years of evolution.

1.1 Background: What Big Data Means After Ten Years

Big data is characterized by the five V's: Volume, Variety, Velocity, Veracity, and Value.

Volume – non‑linear growth of data collection, storage, and computation.

Variety – structured and unstructured data, with rapid growth of multimedia.

Velocity – fast data growth and processing speed, demanding high timeliness.

Veracity – low signal‑to‑noise ratio requiring deep mining.

Value – data as an asset with synergistic benefits.

These trends drive the need for higher compute power, intelligent processing, and ecosystem development.

1.2 MaxCompute Positioning

MaxCompute is a secure, high‑performance, low‑cost, elastic online big‑data service ranging from GB to EB. It offers rich development tools, data import/export solutions, and multiple distributed computing models, supporting SQL, MapReduce, DAG, Graph, and PAI machine‑learning workloads, as well as third‑party engines like Spark and Flink.

1.3 Competitive Analysis

In the cloud data‑warehouse market, MaxCompute ranks first in China and seventh globally according to the Forrester Wave (Q4 2018). Its goal is to become a "carrier‑grade" engine for IoT, log analysis, AI, and other scenarios, demanding high reliability, automation, and security.

2. 2018 Technical Development Overview

MaxCompute focused on core engine enhancements, open platform integration, and new technology fields.

Efficiency Improvements

At the 2018 Cloud Xi conference, MaxCompute doubled its TPC‑BB 100 TB benchmark performance compared to 2017, supporting 20% hardware growth while driving over 70% business growth.

System Openness and Ecosystem Integration

The Cupid unified computing platform now matches EMR Spark benchmarks, supports K8s interfaces, and offers a complete security framework.

Python distributed project MARS was released, gaining 1,200+ stars in two weeks and delivering three‑times the performance of Dask.

Exploring New Areas

Research directions include AdaptiveOperators, Operator Fusion, and ClusteredTable, as well as Auto Data Warehouse, advanced fail‑checking, ML‑based optimizer research, and large‑scale query sub‑graph matching.

Standardization Contributions

MaxCompute joined the TPC committee (2019) and became a PMC for the ORC storage standard and a contributor to the Calcite optimizer project.

3. Core Technology Stack

3.1 Computing Power

The SQL engine processes over 90% of jobs. It consists of a Compiler (full TPC‑DS support), a Runtime (LLVM‑optimized, columnar processing), and an Optimizer (HBO and Calcite‑based CBO).

NewSQL merges declarative and imperative paradigms, adding support for GROUPING SETS, IF‑ELSE, and dynamic type functions.

3.2 Storage

MaxCompute stores EB‑scale data between MaxCompute Tasks and the underlying Pangu distributed file system, evolving from row‑store CFile1 to column‑store CFile2 and a third‑generation format, achieving ~8% storage savings and 20% compute efficiency gains.

Supported compression includes ZSTD with Normal, High, and Extreme strategies.

Features such as Hash Clustering and Range Clustering, along with optimizations like ShuffleRemove and Clustering Pruning, improve query performance.

3.3 System Framework

Resource and task management provides stable job interfaces, short‑job query acceleration, automatic OOM detection, and service‑level flow control.

Metadata services (Catalog, MetaServer, AuthServer) ensure ACID guarantees, high‑availability storage, and high‑QPS authentication, supporting fine‑grained column‑level access control and data classification.

Security combines compute and network virtualization for multi‑tenant isolation, native storage encryption with KMS‑managed keys (BYOK) and AES‑256, with plans for SM algorithms.

3.4 Ecosystem

The Cupid platform enables Spark, Flink, TensorFlow, NumPy, Elasticsearch, and other engines to run on shared data, exposing Yarn/HDFS interfaces and now supporting Kubernetes.

External tables allow seamless integration with OSS, TableStore, TDDL, and Volume, supporting formats like ORC, Parquet, CSV, and JSON, breaking data silos and extending the data lake.

3.5 Intelligence

Auto Data Warehousing leverages historical job features to recommend optimal Hash Clustering strategies, automating table selection, clustering keys, and bucket numbers for massive workloads.

4. Commercial Journey

From its 2009 inception as Cloud Ladder to ODPS and finally MaxCompute, the platform has grown to over 100,000 servers across 18 regions, serving thousands of customers in retail, media, finance, health, education, and more.

MaxCompute now provides batch, streaming, in‑memory, and machine‑learning compute capabilities, forming a critical backbone for Alibaba's digital economy.