Collection size

96 articles

Page 1 of 5

Network Intelligence Research Center (NIRC)

Mar 10, 2025 · Artificial Intelligence

Revisiting Knowledge Distillation for Autoregressive Language Models

The article analyzes why larger teacher models can hurt student performance in autoregressive language model distillation, reveals that different tokens require distinct teaching modes, proposes an Adaptive Token‑wise Knowledge Distillation (ATKD) method, and shows through extensive experiments that ATKD consistently improves accuracy by about 3 % and enhances generalization across model sizes.

Model Compressionadaptive teachingautoregressive language models

0 likes · 9 min read

Revisiting Knowledge Distillation for Autoregressive Language Models

Old Zhang's AI Learning

May 23, 2026 · Artificial Intelligence

Qwopus 3.6‑27B‑v2: Trace‑Inversion Distillation Cuts Token Use by 36% and Boosts Accuracy

The Qwopus 3.6‑27B‑v2 model reconstructs full step‑by‑step reasoning from compressed Claude outputs using a Trace‑Inverter, creates two high‑quality SFT datasets, and achieves 35.9% token savings, a 2.57‑point accuracy gain on MMLU‑Pro, 75.25% success on SWE‑bench, while running on a single consumer‑grade RTX 5090.

GGUFMMLUModel distillation

0 likes · 11 min read

Qwopus 3.6‑27B‑v2: Trace‑Inversion Distillation Cuts Token Use by 36% and Boosts Accuracy

360 Smart Cloud

Dec 3, 2025 · Artificial Intelligence

How Model Distillation Enhances LLM Performance on the TLM Platform

This article explains the TLM large‑model development platform and details how knowledge distillation—using soft labels, temperature scaling, and combined loss functions—compresses teacher models into efficient student models, with practical steps and evaluation on the platform.

AILLMModel distillation

0 likes · 5 min read

How Model Distillation Enhances LLM Performance on the TLM Platform

Network Intelligence Research Center (NIRC)

Dec 30, 2025 · Artificial Intelligence

Bridging Tokenizer Gaps: Cross-Tokenizer Knowledge Distillation at AAAI 2026

This paper introduces SeDi, a semantics‑ and distribution‑aware cross‑tokenizer knowledge distillation framework that aligns teacher and student token spaces via bipartite graph components and top‑K re‑encoding, achieving state‑of‑the‑art performance and lower exposure bias on multiple LLM benchmarks.

AI researchSemantic Alignmentcross-tokenizer distillation

0 likes · 10 min read

Bridging Tokenizer Gaps: Cross-Tokenizer Knowledge Distillation at AAAI 2026

Old Zhang's AI Learning

Mar 12, 2026 · Artificial Intelligence

Distilling Claude Opus 4.6 into Qwen3.5‑27B: High‑Quality Reasoning on a Single RTX 3090

The article details how Claude Opus 4.6's chain‑of‑thought data were used to distill the 27‑billion‑parameter Qwen3.5‑27B model with Unsloth and LoRA, achieving full‑context inference on a single RTX 3090/4090, while outlining performance numbers, hyper‑parameter tips, benchmark gains and the trade‑offs of losing multimodal abilities.

Claude Opus 4.6GPU inferenceLoRA

0 likes · 7 min read

Distilling Claude Opus 4.6 into Qwen3.5‑27B: High‑Quality Reasoning on a Single RTX 3090

Alibaba Cloud Big Data AI Platform

Feb 25, 2025 · Artificial Intelligence

How DistilQwen2.5 Boosts LLM Efficiency with Dual‑Stage Knowledge Distillation

This article introduces DistilQwen2.5, a lightweight LLM series built on Qwen2.5 that uses a novel two‑layer distillation framework, instruction‑data optimization, and parameter‑fusion techniques to achieve higher performance while drastically reducing computational cost and deployment overhead.

Efficient InferenceLLMModel Compression

0 likes · 26 min read

How DistilQwen2.5 Boosts LLM Efficiency with Dual‑Stage Knowledge Distillation

Alibaba Cloud Big Data AI Platform

Apr 22, 2025 · Artificial Intelligence

How DistilQwen2.5-DS3-0324 Achieves Fast, Accurate Reasoning via Quick‑Think Distillation

This article introduces DistilQwen2.5-DS3-0324, a distilled language model series that balances rapid inference with strong reasoning by applying a fast‑thinking chain‑of‑thought strategy, details its two‑stage distillation framework, evaluation on diverse benchmarks, and provides code for downloading and using the models.

Chain of ThoughtDeep LearningModel distillation

0 likes · 17 min read

How DistilQwen2.5-DS3-0324 Achieves Fast, Accurate Reasoning via Quick‑Think Distillation

Alibaba Cloud Big Data AI Platform

Jun 30, 2025 · Artificial Intelligence

Unlocking Small LLM Power: Variable‑Length Chain Distillation with DistillQwen‑ThoughtY

This article introduces a variable‑length chain‑of‑thought distillation technique built on Alibaba Cloud PAI’s EasyDistill toolkit, presents the high‑quality OmniThought‑0528 dataset, details the training of the DistillQwen‑ThoughtY 4B/8B/32B models, and provides code and usage examples for researchers and practitioners.

Chain of ThoughtDatasetDistillation

0 likes · 15 min read

Unlocking Small LLM Power: Variable‑Length Chain Distillation with DistillQwen‑ThoughtY

PaperAgent

Apr 29, 2026 · Artificial Intelligence

Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy

The article introduces the TRS (Thinking with Reasoning Skills) framework, which distills historical LLM reasoning traces into reusable skill cards, enabling offline skill‑base construction and online retrieval that dramatically reduces token consumption (6‑59%) and often improves accuracy on math and coding tasks.

Reasoning SkillsTRScode generation

0 likes · 13 min read

Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy

Baobao Algorithm Notes

Jun 3, 2025 · Artificial Intelligence

Can 1K Fine‑Tuning Replace 100K RL Steps? Insights from Re‑distillation Research

An extensive analysis shows that a 1K‑sample fine‑tuning stage can replicate the generalization gains of thousands of reinforcement‑learning steps, explains the compressibility of RL, introduces a sample‑effect theory, and demonstrates that re‑distillation and small‑scale SFT dramatically improve LLM performance.

Re-distillationReinforcement LearningSample Effect

0 likes · 23 min read

Can 1K Fine‑Tuning Replace 100K RL Steps? Insights from Re‑distillation Research

Machine Heart

Jul 8, 2026 · Artificial Intelligence

How DOPD Overcomes the Privilege Illusion to Boost Online Policy Distillation

The DOPD paper introduces an advantage‑aware dual distillation framework that eliminates the privilege illusion, dynamically selects token‑wise strategies, and delivers up to 7.5‑point gains on LLM benchmarks while closing 89.8% of the teacher‑student gap and showing strong robustness across model sizes.

LLMVLMdual on-policy distillation

0 likes · 9 min read

How DOPD Overcomes the Privilege Illusion to Boost Online Policy Distillation

Tech Freedom Circle

Jul 16, 2026 · Artificial Intelligence

Cut LLM Costs by 500× with Knowledge Distillation – A Step‑by‑Step Guide for Every Company

The article explains why large language model (LLM) inference is prohibitively expensive, outlines the three main drawbacks—high API cost, latency, and hardware requirements—and shows how knowledge distillation can reduce these costs by up to 500×, providing a detailed 7‑step workflow, white‑box vs black‑box methods, code examples, and compliance considerations.

LoRAQLoRAcost reduction

0 likes · 41 min read

Cut LLM Costs by 500× with Knowledge Distillation – A Step‑by‑Step Guide for Every Company

Alibaba Cloud Big Data AI Platform

May 28, 2025 · Artificial Intelligence

How EasyDistill Simplifies LLM Knowledge Distillation for Faster, Smaller Models

EasyDistill, an open‑source toolkit from Alibaba Cloud AI Platform, streamlines knowledge distillation of large language models by offering modular data synthesis, black‑box and white‑box training, reinforcement‑learning and preference‑optimization techniques, enabling the creation of compact, high‑performance DistilQwen models and accompanying datasets.

DistilQwenEasyDistillModel Compression

0 likes · 17 min read

How EasyDistill Simplifies LLM Knowledge Distillation for Faster, Smaller Models

Meituan Technology Team

Jan 8, 2026 · Artificial Intelligence

Must‑Read AAAI 2026 Papers: Efficient Reasoning, Annealing, Multimodal Diffusion & More

This article curates eight AAAI 2026 papers authored by the Meituan research team, covering verifiable stepwise rewards for LLM reasoning, annealing strategies in large‑scale training, process reward models, competence‑difficulty sampling, high‑fidelity visual text rendering, counterfactual fusion, compress‑then‑rank reranking, and cross‑modal quantization for generative recommendation, with direct PDF links for each work.

AAAI2026LLMRecommendation

0 likes · 14 min read

Must‑Read AAAI 2026 Papers: Efficient Reasoning, Annealing, Multimodal Diffusion & More

Machine Learning Algorithms & Natural Language Processing

Jun 18, 2026 · Artificial Intelligence

TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%

The paper introduces Thinking‑Based Non‑Thinking (TNT), a reinforcement‑learning approach that sets a per‑question dynamic token ceiling for non‑thinking mode using the answer length from thinking mode, cutting reward‑hacking incidence to under 10% while boosting accuracy and cutting token usage by nearly half across several math benchmarks.

ACL 2026NLPReinforcement Learning

0 likes · 10 min read

TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%

Machine Heart

Jul 16, 2026 · Artificial Intelligence

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy

Long‑thinking LLMs often waste 41‑52% of tokens after the final answer; the PUMA framework detects when reasoning stops producing new semantic information, enabling early exit that cuts average token usage by 26.2% while keeping accuracy stable and even improving speed across multiple benchmarks.

PUMAearly exitllm-inference

0 likes · 9 min read

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy

Baobao Algorithm Notes

Apr 27, 2025 · Artificial Intelligence

How DeepSeek R1T‑Chimera Cuts Tokens by 40% Without Fine‑Tuning

The DeepSeek‑R1T‑Chimera model merges DeepSeek‑R1 reasoning with V3‑0324 architecture, reusing most V3 weights and swapping only the blue‑highlighted R1 routing experts, achieving the same intelligence as R1 while reducing output tokens by about 40% and running faster, all without any fine‑tuning or distillation.

Artificial IntelligenceDeepSeekLLM

0 likes · 5 min read

How DeepSeek R1T‑Chimera Cuts Tokens by 40% Without Fine‑Tuning

Xiaohongshu Tech REDtech

Nov 4, 2025 · Artificial Intelligence

Unveiling the Law of Capacity Gap: Boosting Language Model Distillation Efficiency

At ACL 2025, a collaborative paper introduced the Law of Capacity Gap, revealing a linear 2.5× optimal teacher‑student size relationship in language model distillation, dramatically cutting compute costs and achieving Pareto‑optimal efficiency, with the MiniMA model as a successful demonstration.

DistillationMiniMAartificial-intelligence

0 likes · 7 min read

Unveiling the Law of Capacity Gap: Boosting Language Model Distillation Efficiency

AntTech

Jun 21, 2025 · Artificial Intelligence

Ring-lite: Open‑Source Lightweight MoE Model Sets SOTA on AIME and LiveCodeBench

Ring-lite, an open‑source lightweight Mixture‑of‑Experts inference model built on Ling‑lite‑1.5, introduces the C3PO reinforcement‑learning training method and achieves state‑of‑the‑art results on benchmarks such as AIME24/25, LiveCodeBench, CodeForce, and GPQA‑diamond, while offering full transparency of weights, code, and data.

AI inferenceC3PObenchmark

0 likes · 11 min read

Ring-lite: Open‑Source Lightweight MoE Model Sets SOTA on AIME and LiveCodeBench

Didi Tech

Mar 12, 2026 · Artificial Intelligence

How STAPO Improves Large‑Model Fine‑Tuning by Silencing Spurious Tokens

The STAPO (Spurious‑Token‑Aware Policy Optimization) algorithm, introduced by Tsinghua University's iDLab and Didi's Deep Sea Lab, tackles policy‑entropy instability and performance oscillation in reinforcement‑learning fine‑tuning of large models by mathematically analyzing token collision probability, defining spurious tokens, and applying a Silencing Spurious Tokens mechanism that yields state‑of‑the‑art results on multiple math‑reasoning benchmarks.

AI safetyFine-tuningReinforcement Learning

0 likes · 7 min read

How STAPO Improves Large‑Model Fine‑Tuning by Silencing Spurious Tokens