How Alibaba’s TePDist Automates Distributed Deep Learning for Large Models

Alibaba Cloud Machine Learning Platform PAI has officially released TePDist, a self‑developed, HLO‑based fully automatic distributed deep‑learning system.

TePDist decouples distributed strategy search from the user’s model‑building language, maintaining generality while automatically exploring high‑performance strategies within acceptable search times, requiring no changes to the model code.

Beyond a distributed compiler, TePDist includes its own runtime to implement automatically searched parallel strategies. It follows a client/server architecture: the server receives HLO IR, explores and applies distributed parallel strategies; the client converts user models into HLO IR, allowing any HLO‑capable client to connect.

Functionally, TePDist consists of two parts: (1) strategy search on HLO IR for SPMD (data parallel and sharding) and pipeline parallelism, building a task‑graph execution plan; (2) an efficient distributed execution engine. It offers multiple optimization levels, with higher levels prioritising strategy quality and lower levels using heuristics for faster search.

Performance tests on GPT and MoE models with mixed SPMD + pipeline strategies show TePDist achieving 62 % and 58 % of peak capability respectively, and its generality is validated on VGG‑19, DNABert and UNet models.

Recognising the industrial potential of large models such as ChatGPT, Alibaba Cloud PAI also announced that TePDist is open‑sourced to help AI developers build faster, better automatic distributed systems.

Open‑source repository: https://github.com/alibaba/TePDist