Can the CaR Method Achieve Better LLM Performance with Only 1.4% of Training Data?

Background

Training large language models (LLMs) requires massive amounts of data, but the quality and diversity of that data are critical factors. While infrastructure upgrades (e.g., MOE load balancing, RoPE, FlashAttention) receive much attention, the role of the data "recipe"—choosing what and how much to train on—remains a decisive, experience‑driven step.

Problem Statement

Practitioners need practical methods to assess data quality and to sample diverse instructions efficiently, especially under limited compute resources. Simple yet effective techniques are sought after, even if they lack the flashiness of advanced model components.

Proposed Solution: CaR Method

The CaR (Clustering and Ranking) method combines two straightforward ideas:

Quality evaluation: Use a scoring model (BERT + regression) to assign a quality score to each instruction.

Diversity selection: Reduce sentence embeddings with PCA, then cluster them using K‑Means (178 clusters). From the full instruction pool, first pick the top‑scoring N1 items, then from each cluster select the top‑scoring N2 items, merge the sets, and remove duplicates.

The authors report that CaR reaches performance comparable to using the full dataset while consuming only 1.96% of the data (approximately 1/70 of the original size).

Technical Details

Quality scoring model: A BERT backbone followed by a regression head was trained on 2,541 labeled instructions. On the held‑out test set, it achieved 84.5% accuracy, outperforming GPT‑3.5‑Turbo (57.48%) and GPT‑4‑1106‑preview (63.19%). The authors caution that the model may be over‑fitted to this small benchmark and that out‑of‑distribution performance remains unverified.

Diversity implementation: Sentence vectors are first reduced by PCA, then clustered with K‑Means (k=178). Selection proceeds in two stages: (1) pick the highest‑scoring N1 instructions from the entire pool; (2) within each of the K clusters, pick the top‑scoring N2 instructions, combine them, and deduplicate.

The paper associated with this work is titled "Clustering and Ranking: Diversity‑preserved Instruction Selection through Expert‑aligned Quality Estimation" and can be accessed at https://arxiv.org/abs/2402.18191. The implementation is open‑source at https://github.com/IronBeliever/CaR.

Results

Empirical evaluation shows that using only 1/70 of the original instruction data, the CaR pipeline achieves better or comparable performance to the baseline that uses the full dataset. The authors note that this method is a baseline many teams have already experimented with, but it remains a solid starting point.

Practical Considerations

In real‑world applications, the CaR pipeline should be adapted to specific scenarios: refine the quality model for the target domain, adjust the number of clusters, and possibly incorporate hierarchical or label‑aware sampling for finer‑grained diversity.

Conclusion

Effective handling of data quality and diversity is a universal lever for improving supervised learning tasks, especially as LLMs scale up. The CaR method demonstrates that a modest fraction of well‑selected data can substantially reduce training cost while preserving performance, highlighting a promising direction for future scaling‑law research.