Alibaba Cloud EMR’s AI Evolution: Accelerating Big Data Performance

EMR Development History Review

Since its first release in 2016, Alibaba Cloud EMR has been built on an open‑source ecosystem, gradually forming a public‑cloud big‑data platform that integrates mainstream open‑source compute and storage engines such as Hadoop, Hive, Spark, and StarRocks. Over nine years, EMR has supported massive data‑processing needs for Alibaba Group’s core businesses (e.g., Taobao Flash Sale, A+) and served public‑cloud customers across e‑commerce, finance, retail, manufacturing, and many other industries. The evolution of EMR moves from a simple open‑source component version to an enterprise‑grade data platform for lake‑warehouse integration and real‑time intelligent scenarios, shifting from “open source” to “high‑efficiency and intelligent”.

New Challenges for Big‑Data Systems in the AI Era

With the rise of large models and generative AI, data‑system boundaries are being redefined. Users now expect to express analysis intent via natural language rather than writing SQL or configuring jobs, and systems must handle multimodal data (streams, text, vectors, semi‑structured logs). Traditional batch, OLAP, machine‑learning, and full‑text search capabilities are required to cooperate on a unified platform. This demands extreme performance, high autonomy, open compatibility, and out‑of‑the‑box usability. Existing big‑data systems face metadata storms, serial I/O, and inefficient reads under a compute‑storage separation architecture, becoming bottlenecks for AI‑driven value extraction.

Efficiency: Out‑of‑the‑Box, Extreme Performance

To address these challenges, EMR on ECS optimized the I/O path across the full stack, tackling three major performance bottlenecks. For metadata storms, a batch‑parallel processing mechanism reduced metadata acquisition from minutes to seconds. For serial compute‑I/O waiting, vectorized asynchronous prefetch and dynamic adaptive read‑ahead enabled parallel compute and data loading. For small‑file and scattered reads, request merging and parallel pre‑open dramatically improved throughput. Benchmarks show a 40% performance gain for TPC‑DS 1 TB queries and up to 90% compute savings in small‑file‑intensive scenarios, delivering true “out‑of‑the‑box” high‑performance experience.

Intelligence: AI Upgrade, High Autonomy

Beyond efficiency, EMR focuses on lowering the usage threshold. The EMR AI Assistant entered public beta, allowing users to ask natural‑language questions such as “Why is the cluster slowing down?” or “Why did the elastic scaling fail at 3 am?” The system automatically analyzes logs, metrics, and execution plans to provide precise diagnostics and remediation suggestions, covering common ECS cluster issues, resource bottlenecks, and performance problems with 24/7 self‑service.

EMR Serverless StarRocks also adds health diagnostics, business insights, event notifications, and an AI Center. It offers T+1 global health assessments, real‑time component fault localization, SQL profiling reports, and business‑impact analytics linking technical metrics to business outcomes.

EMR AI Function: Bringing Large Models to SQL

To bridge data analysis and AI, EMR Serverless StarRocks and Spark launch AI Function beta. Users can invoke large‑model functions directly in SQL for sentiment analysis, sensitive‑information masking, text summarization, translation, ticket classification, etc. Example:

SELECT ai_mask('John Doe lives in New York. His email is [email protected].', ['person', 'email'])

The function returns masked results using Alibaba Cloud’s Baichuan general model, while also supporting custom model integration.

EMR Serverless Spark fully supports GPU scheduling, enabling job‑level GPU allocation, AI Function local inference, and GPU‑accelerated Spark ML (e.g., XGBoost, LightGBM). It integrates with Baichuan, PAI EAS, or private GPU model services to create an end‑to‑end AI data‑processing loop.

Hands‑On Demo

The demo showcases an intelligent multimodal automotive data analysis solution using EMR Serverless Spark and EMR Serverless StarRocks. Spark Notebook processes vehicle annotation data, AI Function extracts driving‑behavior labels from acceleration data, and results are written to StarRocks for multilingual statistical analysis, achieving minute‑level intelligent analysis versus traditional day‑level latency.

The presentation concludes with an invitation to try the latest EMR capabilities.