AntTech
May 24, 2022 · Artificial Intelligence
WPipe: Group‑Based Interleaved Pipeline Parallelism for Large‑Scale DNN Training
The paper introduces WPipe, a group‑based interleaved pipeline parallelism method that reduces memory overhead and weight‑update latency compared with PipeDream‑2BW, achieving up to 1.4× speed‑up and 36% lower memory usage while preserving model accuracy on large‑scale DNNs.
Memory EfficiencyWPipedeep learning
0 likes · 13 min read