How Alibaba’s MaxCompute Tackled Double‑11’s EB‑Scale Data with Fuxi 2.0 and StreamlineX

Background

2019 marked Alibaba’s 11th Double 11 event, during which the e‑commerce system handled over 500,000 transactions per second and generated massive data volumes. MaxCompute’s daily data throughput approached the exabyte (EB) level, with task counts reaching tens of millions.

Challenges

Improving platform performance under ultra‑large‑scale compute and tight resource conditions.

Ensuring stability of disk I/O, data file read/write, and long‑tail job retries under extreme hotspot key and data‑inflation scenarios.

Providing agile, reliable, and efficient distributed job scheduling for nearly ten‑million‑scale jobs.

Guaranteeing resources for high‑priority baseline jobs.

Supporting mixed‑mode (online/offline) workloads on cloud clusters for the first time.

Fuxi 2.0 Scheduler

Fuxi, one of the three original services of the Feitian platform, was designed to solve large‑scale distributed resource scheduling (a multi‑objective optimal matching problem). Fuxi 2.0 is now the built‑in scheduler for MaxCompute, enabling efficient data exchange across tens of thousands of nodes while dynamically adjusting execution plans based on real‑time data distribution.

How the Challenges Were Addressed

Performance optimization with StreamlineX + Shuffle Service automatically matches processing modes and algorithms to real‑time data characteristics, improving average SQL speed by ~20% and reducing error‑retry rates to one‑tenth.

DAG 2.0 provides a more agile scheduling engine and comprehensive de‑blocking capabilities, delivering up to 50% performance gains for large‑scale MapReduce jobs and 20%+ end‑to‑end time reduction for 1 TB TPC‑DS workloads.

Fuxi introduces fine‑grained resource guarantees for high‑priority jobs, including interactive pre‑emptive resource stealing that can allocate resources to critical jobs within 90 seconds.

StreamlineX + Shuffle Service

Streamline (also known as Shuffle or Exchange in other OLAP systems) handles data serialization, compression, transmission, grouping, and sorting between distributed tasks. It accounts for over 30% of SQL runtime and up to 60% in large jobs. StreamlineX (SLX) completely rewrites this component, adding support for GraySort, Adaptive modes, CPU cache‑friendly structures, IO compression, and dynamic memory allocation.

Shuffle Service replaces traditional disk‑based shuffle files, eliminating fragmented reads and improving IO stability. It dynamically schedules data flow, reducing resource consumption by more than 50% compared with classic network shuffle.

Key Technical Improvements

Decoupled runtime Streamline from the Fuxi SDK for better maintainability and extensibility.

GraySort mode: writer groups without sorting, reader performs full sort, boosting pipeline efficiency.

Adaptive mode: runtime can switch between sorted and unsorted processing without extra IO overhead.

CPU cache optimizations reduce cache misses and function‑call overhead.

IO optimizations via multiple compression algorithms and a redesigned shuffle storage format.

Memory optimizations allocate buffers on demand, minimizing dump operations.

Results of StreamlineX + Shuffle Service (Double 11)

Real‑time SQL jobs saw >15% end‑to‑end speedup; offline Streamline throughput increased by 100%.

Effective IO read/write size grew >100%, while disk pressure dropped >20%.

Worker PVC failure probability fell to one‑tenth of its previous value.

High‑priority shuffle traffic was reduced by >25% through resource‑priority guarantees.

DAG 2.0 Overview

DAG 2.0 is the next‑generation directed‑acyclic‑graph execution engine that coordinates resource management, machine management, compute engines, and shuffle components. It enhances dynamic scheduling, fault tolerance, and flexibility to meet the growing diversity of workloads.

DAG 2.0 Key Technologies

Comprehensive error‑handling with machine‑state management and proactive fault isolation.

Conditional join: the optimizer can emit a conditional DAG that selects the best join strategy at runtime based on upstream data size, avoiding OOM and reducing execution time.

DAG 2.0 Double 11 Outcomes

Supported >80% of projects on Double 11; instance overhead reduced by 1–2×.

All high‑priority baseline jobs completed without manual intervention or failures.

Millisecond‑level jobs increased by >20%; resource wait‑time reduced by ~50%.

PAI TensorFlow CPU/GPU jobs migrated to DAG 2.0, ensuring timely model training.

Interactive Pre‑emptive Resource Scheduling

FuxiMaster allocates resources to tasks with micro‑second latency. Interactive pre‑emption allows low‑priority jobs to finish within ~90 seconds instead of being killed, preserving compute resources while delivering resources to high‑priority baseline jobs.

Interactive Pre‑emption Results (Double 11)

High‑priority baseline jobs consistently obtained resources within the expected 90‑second window.

90th‑percentile resource acquisition time for schedule units (SUs) was around one minute across tested clusters.

Overall, the combination of Fuxi 2.0, StreamlineX + Shuffle Service, DAG 2.0, and interactive pre‑emption enabled MaxCompute to process near‑exabyte daily data, run tens of millions of jobs, and maintain stable, efficient operation during the most demanding Double 11 workload.