Why Calibration Data Outperforms Pruning Algorithms in LLM Compression

Findings

Comparison of five recent post‑training pruning algorithms on the open‑source DCLM‑7B model shows two dominant factors: (1) the choice of calibration dataset can improve pruning performance by roughly one percentage point; (2) calibration data has a larger impact than algorithmic refinements, and unsuitable data can nullify the advantage of otherwise competitive pruning methods.

Empirical Study

DCLM‑7B is used because its full pre‑training corpus is publicly available, enabling a controlled analysis of how calibration data affect pruning.

Question 1 – Effect of Calibration Data Across Sparsity Settings

We evaluated sparsity ratios from 50 % to 70 % and both unstructured and semi‑structured (e.g., 2:4, 4:8) pruning types. Using the Wanda algorithm as a representative, we measured the performance gap (max – min) across four calibration sets: C4, Wikipedia, Slimpajama, and the original DCLM pre‑training data. The gap widens as sparsity increases and as the pruning pattern becomes more structured, indicating that the calibration objective dominates when the pruning problem is harder. In extreme cases, a poor calibration set leads to lower commonsense‑reasoning accuracy than a simple magnitude‑based baseline.

Question 2 – Influence of Calibration Data Quantity

For each of the four datasets we sampled calibration sets of size 64, 128, 256, 512, 1024, and 2048 sequences (each 2048 tokens). Across all pruning algorithms the performance variation was minimal, confirming that post‑training pruning is largely insensitive to the amount of calibration data once a modest size (≈128 examples) is reached.

Question 3 – What Makes Good Calibration Data?

Two hypotheses were tested:

Calibration data that are statistically similar to the model’s original training corpus yield better pruning results.

Higher editorial quality of the calibration data improves pruning performance.

We evaluated three public corpora (C4, Slimpajama, Wikipedia) and the DCLM pre‑training set under a 2:4 semi‑structured sparsity of 60 %. All three pruning algorithms (Wanda, SparseGPT, and a reconstruction‑error method) showed the same ranking: DCLM > C4 > Slimpajama > Wikipedia. Despite Wikipedia’s high human‑curated quality, it lagged behind C4 by 1.2–1.5 % absolute performance, suggesting that data quality alone is not decisive.

Qualitative inspection revealed that C4, Slimpajama and DCLM all originate from Common Crawl, whereas Wikipedia is sourced elsewhere, explaining the higher similarity to the training distribution.

Quantitatively, we encoded 3‑grams from each corpus with MinHash‑LSH and computed Jaccard similarity to the DCLM corpus:

C4: 0.07

Slimpajama: 0.016

Wikipedia: 0.008

The similarity scores align with the observed pruning performance ordering.

Synthetic Calibration Data

When the original training corpus is unavailable, we generate synthetic calibration data by prompting the target LLM with a short prefix (the first t tokens) sampled from Wikipedia, letting the model continue generation, and then discarding the top k % highest‑perplexity samples. This filtering mitigates low‑quality generations.

Table 2 (reproduced below) reports pruning results at 60 % sparsity on DCLM‑7B. Synthetic data outperforms all real calibration sets on both language‑model perplexity (Alpaca) and commonsense‑reasoning accuracy (seven benchmark tasks). Notably, synthetic data surpasses the original DCLM training data by 0.4–0.7 % on reasoning tasks, indicating that model‑generated data can be more representative of the patterns needed for accurate importance estimation.

Additional Analyses

We validated the synthetic‑data approach across:

Sparsity levels 50 % and 65 %.

Semi‑structured patterns 4:8 and 2:4.

Other LLM families (e.g., LLaMA).

Varying prefix lengths ( t) and filtering ratios ( k).

All experiments confirm that similarity to the original training distribution remains the primary driver of pruning success.

Related Work Comparison

Prior studies have examined calibration data for quantization and pruning. Shin et al. highlighted over‑fitting of mean‑squared‑error objectives and proposed synthetic data to alleviate it. Bandari et al. compared pre‑training versus downstream data as calibration sources, finding task‑specific data sometimes beneficial. Williams et al. also used synthetic data but generated from start‑of‑sentence tokens; our prefix‑based method is simpler and yields higher similarity to the training corpus.

Conclusion

Calibration data is a decisive factor in post‑training pruning of large language models. Datasets that closely resemble the original pre‑training corpus lead to superior sparsity‑induced performance, and synthetic data generated from the model itself provides a practical substitute when the true corpus is inaccessible.

References

On the Impact of Calibration Data in Post‑training Quantization and Pruning: https://arxiv.org/abs/2311.09755

Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization: https://arxiv.org/abs/2406.15524

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning: https://arxiv.org/abs/2410.07461

Self‑calibration for Language Model Quantization and Pruning: https://arxiv.org/abs/2410.17170