How TAPIR Boosts Small LLMs with Task‑Aware Curriculum Planning

The paper introduces TAPIR, a task‑aware curriculum planning framework that distills instruction‑following abilities from black‑box LLM teachers into smaller student models by filtering difficult prompts, resampling tasks, enhancing response styles, and iteratively optimizing across multiple training rounds, achieving superior performance on benchmark evaluations.

Instruction TuningLLM distillationTAPIR

0 likes · 10 min read

How TAPIR Boosts Small LLMs with Task‑Aware Curriculum Planning

DataFunTalk

Feb 2, 2024 · Artificial Intelligence

Utilizing Negative Samples for Knowledge Distillation of Large Language Models

This paper presents a novel framework that leverages negative samples during large language model distillation through three stages—Negative Assistive Training, Negative Calibration Enhancement, and Adaptive Self‑Consistency—demonstrating significant accuracy gains on challenging mathematical reasoning benchmarks and improved generalization to out‑of‑distribution tasks.

Chain-of-ThoughtKnowledge TransferLLM distillation

0 likes · 13 min read

Utilizing Negative Samples for Knowledge Distillation of Large Language Models