How MaxCompute Evolves Big Data Platforms for AI: Architecture, Core Capabilities, and Real‑World Cases

Key Capabilities of MaxCompute Data+AI

MaxCompute, Alibaba Cloud's core big‑data compute platform, is evolving around two pillars—storage and computing—to meet AI demands. Its architecture is divided into data, model, compute, and engine layers.

Data Layer

MaxCompute stores both structured and unstructured data and supports multimodal BLOB fields for audio, video, and other media. It integrates with OSS, Hologres, and other external storage engines via Object Table and external storage access.

Model Layer

The platform hosts traditional machine‑learning models such as XGBoost and LightGBM, as well as open‑source large models like Qwen and DeepSeek, and can invoke commercial flagship models from the Bailei platform for unified model management.

Compute Layer

MaxCompute provides mixed CPU and GPU scheduling. Users declare required resources declaratively, enabling heterogeneous compute for multimodal data processing.

Engine Layer

Two core compute interfaces are offered: a SQL engine with SQL AI functions that call large models for offline inference, and MaxFrame , a native Python distributed‑compute framework designed for data scientists and AI engineers.

Storage Management for AI

MaxCompute has progressed from data federation and lake‑warehouse integration to AI‑centric multimodal data management. Its "lake‑warehouse multimodal data unified management" architecture stores images, video, audio, etc., in internal tables (BLOB) or external storage (OSS) without moving data, using the unified metadata service Max Meta and Storage API .

It supports a single‑table, multi‑column mixed storage model, allowing multimodal content and associated metadata (e.g., Prompt) to reside together in BLOB or JSON columns, simplifying AI inference and AIGC data organization.

Compute Engine for AI

MaxCompute’s engine focuses on model management, AI functions, and heterogeneous computing.

1. SQL AI Function

SQL analysts can invoke large models directly from SQL, lowering the barrier for AI inference on structured data.

2. MaxFrame – Cloud‑Native Distributed Python Framework

MaxFrame is compatible with Pandas, XGBoost, LightGBM, etc., and runs distributed operators on MaxCompute’s massive compute resources. It integrates with MaxCompute Notebook, image management, and provides a seamless Python development ecosystem.

Key advantages:

Heterogeneous mixed scheduling : CPU (CU) and GPU (GU) resources can be specified programmatically.

Distributed data‑processing operators : Supports open‑source libraries and automatic distributed execution.

Stable, convenient development experience : Deep integration with DataWorks, custom image support, OSS mounting, and AI assistant for faster development.

3. MaxFrame AI Function

Built‑in AI Function integrates large models such as Qwen‑3 (8B/14B/32B) and DeepSeek‑R1‑Distill‑Qwen. Users specify the model, prompts, and parameters via SDK to run large‑scale offline inference on both structured and multimodal data.

4. Convenient Development Experience

MaxFrame SDK is publicly available and can be used in VS Code, Jupyter Notebook (via pip install maxframe), DataWorks Notebook (via Magic Command), and DataWorks data‑development nodes (PyODPS3).

Key Scenarios and Cases

1. Large‑Model Data Pre‑Processing

A leading large‑model company needed PB‑scale storage, >100k cores, secure data governance, and pipeline orchestration. Using MaxFrame, they achieved a >50% performance boost for the MinHash operator, ran stable tasks with 300 k cores, and elastically scaled to 160 k cores, dramatically shortening PB‑scale processing cycles.

2. Automotive Embodied‑Intelligence Data Pre‑Processing

In autonomous‑driving scenarios, massive multimodal data (images, video, radar, GPS) are stored as ROS bags. MaxFrame provided elastic compute and distributed processing, improving data‑processing efficiency by over 40% compared with single‑machine Python.

3. Multimodal Data Processing

Customers in the car‑networking domain required unified management of images, video, and structured data, as well as a Python framework that could invoke third‑party models. MaxFrame’s Object Table enabled unified metadata queries, while built‑in MinHash and custom images (e.g., yolo11n) allowed flexible concurrency and higher job throughput.

4. Typical Cases

• A top‑tier large‑model firm replaced a local solution with MaxCompute + MaxFrame for FastText classification, MinHash deduplication, and CI/CD orchestration in DataWorks. • Automotive autonomous‑driving pipelines leveraged MaxFrame to process BAG‑format data, achieving >40% performance gains over single‑node Python. • Multimodal image tagging combined AI Function with embedding generation for downstream retrieval and analysis.

Conclusion

MaxCompute builds a full‑link Data+AI capability from storage to SQL to Python, forming a cloud‑native, elastic, high‑performance foundation for AI data assets. Whether for large‑model preprocessing or embodied‑intelligence multimodal governance, MaxCompute continuously empowers intelligent transformation across industries.