Collection size

94 articles

Page 1 of 5

Network Intelligence Research Center (NIRC)

Mar 10, 2025 · Artificial Intelligence

Revisiting Knowledge Distillation for Autoregressive Language Models

The article analyzes why larger teacher models can hurt student performance in autoregressive language model distillation, reveals that different tokens require distinct teaching modes, proposes an Adaptive Token‑wise Knowledge Distillation (ATKD) method, and shows through extensive experiments that ATKD consistently improves accuracy by about 3 % and enhances generalization across model sizes.

adaptive teachingautoregressive language modelsknowledge distillation

0 likes · 9 min read

Revisiting Knowledge Distillation for Autoregressive Language Models

Old Zhang's AI Learning

May 23, 2026 · Artificial Intelligence

Qwopus 3.6‑27B‑v2: Trace‑Inversion Distillation Cuts Token Use by 36% and Boosts Accuracy

The Qwopus 3.6‑27B‑v2 model reconstructs full step‑by‑step reasoning from compressed Claude outputs using a Trace‑Inverter, creates two high‑quality SFT datasets, and achieves 35.9% token savings, a 2.57‑point accuracy gain on MMLU‑Pro, 75.25% success on SWE‑bench, while running on a single consumer‑grade RTX 5090.

GGUFMMLUQwen

0 likes · 11 min read

Qwopus 3.6‑27B‑v2: Trace‑Inversion Distillation Cuts Token Use by 36% and Boosts Accuracy

360 Smart Cloud

Dec 3, 2025 · Artificial Intelligence

How Model Distillation Enhances LLM Performance on the TLM Platform

This article explains the TLM large‑model development platform and details how knowledge distillation—using soft labels, temperature scaling, and combined loss functions—compresses teacher models into efficient student models, with practical steps and evaluation on the platform.

AILLMMachine Learning

0 likes · 5 min read

How Model Distillation Enhances LLM Performance on the TLM Platform

Machine Learning Algorithms & Natural Language Processing

Oct 11, 2026 · Artificial Intelligence

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.

DeepSeekGRPOKV-Cache Optimization

0 likes · 37 min read

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

Network Intelligence Research Center (NIRC)

Dec 30, 2025 · Artificial Intelligence

Bridging Tokenizer Gaps: Cross-Tokenizer Knowledge Distillation at AAAI 2026

This paper introduces SeDi, a semantics‑ and distribution‑aware cross‑tokenizer knowledge distillation framework that aligns teacher and student token spaces via bipartite graph components and top‑K re‑encoding, achieving state‑of‑the‑art performance and lower exposure bias on multiple LLM benchmarks.

AI ResearchSemantic Alignmentcross-tokenizer distillation

0 likes · 10 min read

Bridging Tokenizer Gaps: Cross-Tokenizer Knowledge Distillation at AAAI 2026

Old Zhang's AI Learning

Mar 12, 2026 · Artificial Intelligence

Distilling Claude Opus 4.6 into Qwen3.5‑27B: High‑Quality Reasoning on a Single RTX 3090

The article details how Claude Opus 4.6's chain‑of‑thought data were used to distill the 27‑billion‑parameter Qwen3.5‑27B model with Unsloth and LoRA, achieving full‑context inference on a single RTX 3090/4090, while outlining performance numbers, hyper‑parameter tips, benchmark gains and the trade‑offs of losing multimodal abilities.

Claude Opus 4.6GPU InferenceLoRA

0 likes · 7 min read

Distilling Claude Opus 4.6 into Qwen3.5‑27B: High‑Quality Reasoning on a Single RTX 3090

Alibaba Cloud Big Data AI Platform

Feb 25, 2025 · Artificial Intelligence

How DistilQwen2.5 Boosts LLM Efficiency with Dual‑Stage Knowledge Distillation

This article introduces DistilQwen2.5, a lightweight LLM series built on Qwen2.5 that uses a novel two‑layer distillation framework, instruction‑data optimization, and parameter‑fusion techniques to achieve higher performance while drastically reducing computational cost and deployment overhead.

LLMefficient inferenceknowledge distillation

0 likes · 26 min read

How DistilQwen2.5 Boosts LLM Efficiency with Dual‑Stage Knowledge Distillation

Alibaba Cloud Big Data AI Platform

Apr 22, 2025 · Artificial Intelligence

How DistilQwen2.5-DS3-0324 Achieves Fast, Accurate Reasoning via Quick‑Think Distillation

This article introduces DistilQwen2.5-DS3-0324, a distilled language model series that balances rapid inference with strong reasoning by applying a fast‑thinking chain‑of‑thought strategy, details its two‑stage distillation framework, evaluation on diverse benchmarks, and provides code for downloading and using the models.

Large Language Modelschain-of-thoughtdeep learning

0 likes · 17 min read

How DistilQwen2.5-DS3-0324 Achieves Fast, Accurate Reasoning via Quick‑Think Distillation

Alibaba Cloud Big Data AI Platform

Jun 30, 2025 · Artificial Intelligence

Unlocking Small LLM Power: Variable‑Length Chain Distillation with DistillQwen‑ThoughtY

This article introduces a variable‑length chain‑of‑thought distillation technique built on Alibaba Cloud PAI’s EasyDistill toolkit, presents the high‑quality OmniThought‑0528 dataset, details the training of the DistillQwen‑ThoughtY 4B/8B/32B models, and provides code and usage examples for researchers and practitioners.

LLMModel Trainingchain-of-thought

0 likes · 15 min read

Unlocking Small LLM Power: Variable‑Length Chain Distillation with DistillQwen‑ThoughtY

PaperAgent

Apr 29, 2026 · Artificial Intelligence

Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy

The article introduces the TRS (Thinking with Reasoning Skills) framework, which distills historical LLM reasoning traces into reusable skill cards, enabling offline skill‑base construction and online retrieval that dramatically reduces token consumption (6‑59%) and often improves accuracy on math and coding tasks.

Large Language ModelsReasoning SkillsTRS

0 likes · 13 min read

Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy

Baobao Algorithm Notes

Jun 3, 2025 · Artificial Intelligence

Can 1K Fine‑Tuning Replace 100K RL Steps? Insights from Re‑distillation Research

An extensive analysis shows that a 1K‑sample fine‑tuning stage can replicate the generalization gains of thousands of reinforcement‑learning steps, explains the compressibility of RL, introduces a sample‑effect theory, and demonstrates that re‑distillation and small‑scale SFT dramatically improve LLM performance.

Large Language ModelsRe-distillationSample Effect

0 likes · 23 min read

Can 1K Fine‑Tuning Replace 100K RL Steps? Insights from Re‑distillation Research

Machine Learning Algorithms & Natural Language Processing

Oct 26, 2026 · Artificial Intelligence

On-Policy Distillation Explained: White-Box, Self-Distillation & Black-Box Methods

This article provides a comprehensive analysis of On-Policy Distillation (OPD), covering white-box distillation with token-level probability matching, self-distillation using privileged context, black-box distillation via reward modeling, and industrial applications from MiMo, DeepSeek, and Qwen, while discussing OPD's limitations and generalization capabilities.

Black-Box DistillationGKDLarge Language Models

0 likes · 56 min read

On-Policy Distillation Explained: White-Box, Self-Distillation & Black-Box Methods

Machine Heart

Jul 8, 2026 · Artificial Intelligence

How DOPD Overcomes the Privilege Illusion to Boost Online Policy Distillation

The DOPD paper introduces an advantage‑aware dual distillation framework that eliminates the privilege illusion, dynamically selects token‑wise strategies, and delivers up to 7.5‑point gains on LLM benchmarks while closing 89.8% of the teacher‑student gap and showing strong robustness across model sizes.

LLMVLMdual on-policy distillation

0 likes · 9 min read

How DOPD Overcomes the Privilege Illusion to Boost Online Policy Distillation

Data STUDIO

Sep 11, 2026 · Artificial Intelligence

Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression

This article details how a custom C inference engine runs the 744B-parameter GLM 5.2 MoE model on 25GB RAM by quantizing routed experts to int4, streaming them from disk via pread, compressing KV cache 57x with Multi-head Latent Attention, and employing a five-tier memory hierarchy with speculative decoding.

AVX2 kernelsC inference engineGLM-5.2

0 likes · 48 min read

Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression

Tech Freedom Circle

Jul 16, 2026 · Artificial Intelligence

Cut LLM Costs by 500× with Knowledge Distillation – A Step‑by‑Step Guide for Every Company

The article explains why large language model (LLM) inference is prohibitively expensive, outlines the three main drawbacks—high API cost, latency, and hardware requirements—and shows how knowledge distillation can reduce these costs by up to 500×, providing a detailed 7‑step workflow, white‑box vs black‑box methods, code examples, and compliance considerations.

Large Language ModelsLoRAQLoRA

0 likes · 41 min read

Cut LLM Costs by 500× with Knowledge Distillation – A Step‑by‑Step Guide for Every Company

Alibaba Cloud Big Data AI Platform

May 28, 2025 · Artificial Intelligence

How EasyDistill Simplifies LLM Knowledge Distillation for Faster, Smaller Models

EasyDistill, an open‑source toolkit from Alibaba Cloud AI Platform, streamlines knowledge distillation of large language models by offering modular data synthesis, black‑box and white‑box training, reinforcement‑learning and preference‑optimization techniques, enabling the creation of compact, high‑performance DistilQwen models and accompanying datasets.

DistilQwenEasyDistillknowledge distillation

0 likes · 17 min read

How EasyDistill Simplifies LLM Knowledge Distillation for Faster, Smaller Models

Meituan Technology Team

Jan 8, 2026 · Artificial Intelligence

Must‑Read AAAI 2026 Papers: Efficient Reasoning, Annealing, Multimodal Diffusion & More

This article curates eight AAAI 2026 papers authored by the Meituan research team, covering verifiable stepwise rewards for LLM reasoning, annealing strategies in large‑scale training, process reward models, competence‑difficulty sampling, high‑fidelity visual text rendering, counterfactual fusion, compress‑then‑rank reranking, and cross‑modal quantization for generative recommendation, with direct PDF links for each work.

AAAI2026LLMcounterfactual

0 likes · 14 min read

Must‑Read AAAI 2026 Papers: Efficient Reasoning, Annealing, Multimodal Diffusion & More

AndroidPub

Aug 31, 2026 · Artificial Intelligence

Mastering LLM Knowledge Distillation: Theory, DeepSeek Practice & PyTorch Implementation

This article explains knowledge distillation for large language models, comparing compression techniques, detailing target and feature distillation mechanisms, showcasing DeepSeek's distillation of 671B models into smaller Qwen and LLaMA variants, and providing two practical implementation paths: instruction distillation via API and classic logits-based PyTorch code with training tips.

DeepSeekLarge Language ModelsLoRA

0 likes · 20 min read

Mastering LLM Knowledge Distillation: Theory, DeepSeek Practice & PyTorch Implementation

Machine Learning Algorithms & Natural Language Processing

Jun 18, 2026 · Artificial Intelligence

TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%

The paper introduces Thinking‑Based Non‑Thinking (TNT), a reinforcement‑learning approach that sets a per‑question dynamic token ceiling for non‑thinking mode using the answer length from thinking mode, cutting reward‑hacking incidence to under 10% while boosting accuracy and cutting token usage by nearly half across several math benchmarks.

ACL 2026NLPReward Hacking

0 likes · 10 min read

TNT: Dynamic Token Limits Slash Reward Hacking in Mixed Inference Models Below 10%

Machine Heart

Jul 16, 2026 · Artificial Intelligence

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy

Long‑thinking LLMs often waste 41‑52% of tokens after the final answer; the PUMA framework detects when reasoning stops producing new semantic information, enabling early exit that cuts average token usage by 26.2% while keeping accuracy stable and even improving speed across multiple benchmarks.

LLM inferencePUMAearly exit

0 likes · 9 min read

Brake Overthinking in Long‑Reasoning Models by Detecting Semantic Redundancy