WPipe: Group‑Based Interleaved Pipeline Parallelism for Large‑Scale DNN Training

The paper introduces WPipe, a group‑based interleaved pipeline parallelism method that reduces memory overhead and weight‑update latency compared with PipeDream‑2BW, achieving up to 1.4× speed‑up and 36% lower memory usage while preserving model accuracy on large‑scale DNNs.

Deep LearningPipeline ParallelismTraining Throughput

0 likes · 13 min read

WPipe: Group‑Based Interleaved Pipeline Parallelism for Large‑Scale DNN Training