How DeepRec Extension Boosts Distributed Sparse Model Training with Elasticity and Fault Tolerance

Introduction

DeepRec Extension builds on the DeepRec training and inference framework to improve the efficiency of distributed training for large‑scale sparse models. It introduces automatic elastic training, distributed fault tolerance, and resource‑aware scheduling to reduce cost and increase throughput.

Motivation

Sparse models can reach hundreds of GB to TB in size, making distributed training essential. Existing TensorFlow Parameter Server (PS) solutions suffer from complex partitioning, difficult resource estimation, simplistic fault‑tolerance, and slow node handling.

Key Challenges

Complex distributed modeling interfaces lead to uneven parameter placement and reduced throughput.

Resource estimation is hard; static allocation causes OOM or waste.

Fault‑tolerance relies on full checkpoint restores, incurring high latency.

Slow or failed PS nodes degrade overall training speed.

Design Overview

DeepRec Extension decouples from specific frameworks using Operation abstraction, GraphOptimization, and Hook mechanisms. It extends the Kubernetes‑based TFJob CRD with an AIMaster node that manages elastic training, resource monitoring, and automatic backup.

Core Features

Distributed training resource estimation.

Automatic elastic training for PS and Worker roles.

Real‑time resource and graph monitoring (Gazer).

Automatic backup and fault‑tolerance for PS parameters and data.

Elastic Training Mechanism

AIMaster launches an AIMaster pod and service, then creates a TFJob. It estimates initial resources, monitors node status, and triggers elastic scaling without restarting worker or PS processes. Elastic GRPCServer updates cluster membership dynamically.

Fault Tolerance

DataManager tracks global sample state; checkpoints are saved incrementally. When a Worker fails, it is relaunched without state loss. When a PS fails, remaining PS nodes provide parameter backups, allowing rapid recovery without full checkpoint reads.

Performance Results

Automatic elastic training reduces resource waste by up to 42% and improves E2E throughput. Incremental checkpoint backup cuts PS recovery time, especially when network bandwidth is limited.

Future Work

Support joint PS‑Worker elasticity for finer resource utilization.

Develop adaptive parameter migration strategies for hot‑cold sparse data.

Combine fault tolerance with incremental checkpoints to further lower overhead.

DeepRec Extension is open‑source on GitHub, allowing developers to deploy, extend, and customize monitoring and elasticity strategies.