Alibaba Cloud Big Data AI Platform

Apr 24, 2023 · Artificial Intelligence

How Alibaba’s TePDist Automates Distributed Deep Learning for Large Models

Alibaba Cloud’s PAI platform unveils TePDist, an HLO‑based automatic distributed deep‑learning system that decouples strategy search from model code, offers client/server architecture, supports SPMD and pipeline parallelism, delivers high performance on GPT, MoE and other models, and is now open‑source.