How Dataset Distillation Shrinks Training Data Without Losing Accuracy

Background

Deep learning’s rapid adoption has created a tension between ever‑growing data volumes and limited computational resources. Storing, transmitting, and training on massive datasets become costly, motivating research into lightweight data‑processing paradigms.

What Is Dataset Distillation?

Dataset Distillation (DD), also called Dataset Condensation, aims to compress a large training set T into a much smaller synthetic set S (|S| ≪ |T|) such that a model trained on S achieves performance comparable to training on T. Unlike traditional coreset selection, which picks a subset of real samples, DD synthesizes new data points that maximize information density.

Methodological Evolution

1. Meta‑Learning Methods

Early DD work treats the synthetic set S as learnable parameters and formulates a bilevel optimization: the inner loop trains a model on S, the outer loop evaluates the model on the original data T and back‑propagates the “meta‑loss” to update S. This approach yields high‑quality distilled data but incurs prohibitive memory and compute costs because the inner training must be unrolled.

2. Parameter‑Matching Methods

To avoid the expensive bilevel loop, parameter‑matching aligns the dynamics of training on S and T. Two main branches exist:

Gradient Matching : forces the gradient computed on S to match the gradient on T for a given model state.

Trajectory Matching : matches the entire weight trajectory after several training steps, reducing accumulated error.

These methods eliminate higher‑order gradients, improving efficiency, yet still require repeated simulation of training dynamics.

3. Distribution‑Matching Methods

Distribution matching replaces dynamic simulation with a single‑level optimization. A frozen feature extractor ψ maps data to a latent space; the objective minimizes the distance between feature statistics (e.g., class‑wise means) of S and T. This yields the fastest training but may sacrifice a bit of performance compared with parameter‑matching.

4. Factorization & Generative Parameterization

These orthogonal approaches focus on how to represent S. Instead of optimizing every pixel, factorization decomposes S into a basis matrix A and coefficient matrix M ( S = A·M), or learns a latent code b decoded by a small network h. Generative parameterization leverages a pre‑trained generator G (GAN or diffusion) and optimizes only the latent code Z. Both strategies dramatically increase compression ratios and often achieve state‑of‑the‑art results.

Evaluation of Dataset Distillation

Effective DD algorithms are judged on three intertwined dimensions:

Performance : train a model from scratch on S and measure test accuracy (or task‑specific metrics) on the original test set.

Efficiency & Scalability : report training acceleration, total runtime, peak GPU memory, and ability to handle large‑scale datasets such as ImageNet.

Generalization & Transferability : assess cross‑architecture performance, downstream task utility (continual learning, NAS, federated learning), and robustness aspects (privacy, adversarial resistance, backdoor susceptibility).

Practical Applications

Beyond pure efficiency, DD serves several emerging use‑cases:

Continual Learning : distilled data act as a compact replay buffer, mitigating catastrophic forgetting.

Federated Learning : clients upload tiny synthetic datasets instead of raw gradients, reducing communication overhead and enhancing privacy.

Neural Architecture Search : a proxy distilled set enables rapid evaluation of many architectures with comparable ranking to full‑data training.

Privacy‑Preserving AI : synthetic samples obscure raw pixel values; combined with differential privacy they enable safe data sharing in sensitive domains such as medical imaging.

Security & Explainability : distilled data can be used for adversarial training, backdoor detection, and as a human‑interpretable “bridge” to understand which training patterns drive model decisions.

Future Directions

The field is moving toward multimodal distillation—graph, text, and video data—by extending the core ideas to structured and sequential domains. Emerging works explore hybrid pipelines that combine factorization, generative priors, and distribution matching to achieve higher compression while preserving fidelity across modalities.

References

Wang, T. et al., “Dataset distillation,” arXiv:1811.10959, 2018.

Zhao, B. et al., “Dataset condensation with gradient matching,” arXiv:2006.05929, 2020.

Cazenavette, G. et al., “Dataset distillation by matching training trajectories,” CVPR, 2022.

Zhao, B. & Bilen, H., “Dataset condensation with distribution matching,” CVPRW, 2023.

Deng, Z. & Russakovsky, O., “Remember the past: Distilling datasets into addressable memories,” NeurIPS, 2022.

Liu, S. et al., “Dataset distillation via factorization,” NeurIPS, 2022.

Zhao, B. & Bilen, H., “Synthesizing informative training samples with GAN,” arXiv:2204.07513, 2022.

Cazenavette, G. et al., “Generalizing dataset distillation via deep generative prior,” CVPR, 2023.

Wang, K. et al., “CAFE: Learning to condense dataset by aligning features,” CVPR, 2022.

Lei, S. & Tao, D., “A comprehensive survey of dataset distillation,” IEEE TPAMI, 2023.

Liu, P. & Du, J., “The evolution of dataset distillation: Toward scalable and generalizable solutions,” arXiv:2502.05673, 2025.