Negative Sample Assisted Distillation for Large Language Models

Large language models (LLMs) achieve strong performance on many reasoning tasks, but their black‑box nature and massive parameter count hinder practical deployment, especially for complex mathematical problems where they sometimes generate incorrect reasoning chains. Existing work mainly transfers knowledge from positive (correct) samples and ignores synthetic data containing wrong answers.

At AAAI 2024, the Xiaohongshu Search Algorithm team introduced an innovative framework that fully exploits negative samples—data that fail to produce correct answers—during the distillation of LLM reasoning abilities. The framework consists of three serialized steps: Negative Assistance Training (NAT), Negative Calibration Enhancement (NCE), and Adaptive Self‑Consistency (ASC), covering the entire pipeline from training to inference.

The paper demonstrates that negative samples contain valuable knowledge that can improve model specialization. NAT introduces a dual‑LoRA architecture to absorb knowledge from both positive and negative data, dynamically integrating negative LoRA into the positive LoRA training. NCE uses negative knowledge as a baseline to calibrate self‑enhancement, weighting critical reasoning steps more heavily. ASC replaces uniform voting in self‑consistency with a ranking model trained on both positive and negative samples, adaptively re‑weighting candidate reasoning chains during inference.

Experiments were conducted on the challenging MATH benchmark (12,500 problems across seven subjects) and four out‑of‑distribution datasets (GSM8K, ASDiv, MultiArith, SVAMP). Teacher models were GPT‑3.5‑Turbo and GPT‑4; the student model was LLaMA‑7B. Compared with baselines such as fine‑tuning, few‑shot prompting, CoT knowledge distillation, and naive mixing of positive/negative data, the proposed framework consistently achieved higher accuracy. NAT improved task accuracy across all baselines, yielding up to a 75.75 % gain over raw fine‑tuning. NCE added an average 10 % boost over standard knowledge distillation, while ASC outperformed traditional self‑consistency and weighted self‑consistency in answer aggregation.

The study concludes that leveraging negative samples is an effective way to transfer complex reasoning capabilities from large models to compact, specialized models, reducing token consumption and improving performance on both in‑domain and out‑of‑domain tasks.

Paper: https://arxiv.org/abs/2312.12832