Revamper: An Intelligent Data Augmentation Engine for Faster DNN Training

The paper introduces a novel sample‑reuse mechanism called data refurbishing , which splits the traditional augmentation pipeline into partial augmentation (cached) and final augmentation (applied each epoch). By reusing partially augmented samples, the method retains the diversity benefits of augmentation while eliminating most of the CPU‑intensive random transforms.

To evaluate the impact of augmentation overhead, the authors measured training throughput of ResNet‑50 on ImageNet while varying the number of RandAugment layers. Adding more RandAugment layers dramatically reduced throughput, confirming that multi‑layer augmentation is a CPU bottleneck.

Two prior approaches are discussed: (1) GPU/FPGA‑based pipelines such as NVIDIA DALI and TrainBox, which struggle to exploit massive parallelism due to the randomness of augmentation, and (2) Google’s data echoing , which caches fully augmented samples but harms randomness and thus generalization.

Data refurbishing addresses both issues. It introduces a reuse factor r (how many times a cached sample is reused) and a split strategy that determines which augmentations belong to the partial stage. The authors illustrate a 3‑D plot of reuse factor, the ratio of final‑to‑full transforms, and normalized expected unique samples, showing a “sweet spot” where throughput and sample diversity are jointly maximized.

The implementation, named Revamper , replaces standard data loaders (e.g., PyTorch DataLoader and tf.data) and adds three key components:

Balanced eviction : before each epoch, the evict shuffler selects N/r indices to evict, ensuring each cached sample is reused exactly r times and that CPU load is evenly distributed across epochs.

Cache‑aware shuffle : the batch shuffler assigns mini‑batch indices so that each batch contains a constant ratio of cached to non‑cached samples, eliminating idle time between CPU and accelerator.

Cache storage for partially augmented samples and a modified batch shuffler that respects the cache‑aware policy.

Experimental evaluation uses PyTorch 1.6 on ImageNet with RandAugment‑augmented ResNet‑50. Three baselines are compared: (a) standard full augmentation, (b) data echoing (full‑sample caching), and (c) a simplified setting that removes some augmentation layers. Revamper achieves higher training throughput than the standard baseline while maintaining comparable validation accuracy; the data‑echoing and simplified baselines improve throughput but suffer noticeable accuracy drops.

Results show the greatest throughput gains when CPU resources are scarce, confirming that Revamper effectively mitigates the CPU bottleneck of data augmentation without compromising model generalization.

Reference: Refurbish Your Training Data: Reusing Partially Augmented Samples for Faster Deep Neural Network Training .