BestHub
Sign in
Collection size
96 articles
Page 2 of 5
Didi Tech
Didi Tech
Mar 12, 2026 · Artificial Intelligence

How STAPO Improves Large‑Model Fine‑Tuning by Silencing Spurious Tokens

The STAPO (Spurious‑Token‑Aware Policy Optimization) algorithm, introduced by Tsinghua University's iDLab and Didi's Deep Sea Lab, tackles policy‑entropy instability and performance oscillation in reinforcement‑learning fine‑tuning of large models by mathematically analyzing token collision probability, defining spurious tokens, and applying a Silencing Spurious Tokens mechanism that yields state‑of‑the‑art results on multiple math‑reasoning benchmarks.

AI safetySTAPOfine-tuning
0 likes · 7 min read
How STAPO Improves Large‑Model Fine‑Tuning by Silencing Spurious Tokens
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 26, 2026 · Artificial Intelligence

Distilling Claude Opus into Qwen3.6-27B – GGUF Lets You Run Locally on Consumer GPUs

The preview model Qwopus3.6-27B‑v1, distilled from Claude Opus onto Qwen3.6‑27B using SFT with the Unsloth stack and a curated 12 K high‑quality inference sample set, is evaluated on agentic reasoning, front‑end design, and Canvas/WebGL tasks with an RTX 5090, and can be deployed locally via llama.cpp GGUF quantizations with detailed memory guidelines.

Apache 2.0Claude OpusGGUF
0 likes · 7 min read
Distilling Claude Opus into Qwen3.6-27B – GGUF Lets You Run Locally on Consumer GPUs
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Mar 29, 2025 · Artificial Intelligence

How DistilQwen2.5‑R1 Boosts Small‑Model Reasoning with Innovative Knowledge Distillation

The article introduces the DistilQwen2.5‑R1 series, which leverages a novel knowledge‑distillation pipeline—including CoT data evaluation, improvement, and validation—to transfer deep reasoning abilities from large models like DeepSeek‑R1 to compact models, achieving superior performance across math, code, and scientific benchmarks and providing open‑source checkpoints and deployment guides for practical use.

AI inferenceBenchmark EvaluationKnowledge Distillation
0 likes · 17 min read
How DistilQwen2.5‑R1 Boosts Small‑Model Reasoning with Innovative Knowledge Distillation
Architect Practice
Architect Practice
Jul 29, 2026 · Artificial Intelligence

How a 1.5B Model Beats Cutting‑Edge Large Models on Math Exams

This article explains why and how knowledge distillation lets a 1.5 B parameter model surpass much larger LLMs on math benchmarks, detailing the underlying soft‑label transfer, temperature tuning, various distillation families, engineering pipelines, and the practical trade‑offs that bound its success.

AI engineeringKnowledge DistillationLarge Language Models
0 likes · 14 min read
How a 1.5B Model Beats Cutting‑Edge Large Models on Math Exams
AntTech
AntTech
Apr 23, 2026 · Artificial Intelligence

Ling-2.6-flash: Faster Response, Stronger Execution, and Higher Token Efficiency for Agent Workloads

Ling-2.6-flash is a 104B‑parameter Instruct model that uses a mixed‑linear architecture and token‑efficiency optimizations to achieve up to 340 tokens/s inference speed, 4× higher throughput than comparable models, and ten‑fold lower token consumption on Agent benchmarks, while maintaining SOTA performance.

Agent optimizationLLMbenchmark
0 likes · 15 min read
Ling-2.6-flash: Faster Response, Stronger Execution, and Higher Token Efficiency for Agent Workloads
DataFunTalk
DataFunTalk
Oct 30, 2025 · Artificial Intelligence

How On-Policy Distillation Cuts LLM Training Cost by 90%

Thinking Machines Lab introduces On-Policy Distillation, a post‑training technique that matches reinforcement‑learning performance while reducing compute cost by up to tenfold, and demonstrates its effectiveness through extensive experiments on reasoning, personalization, and catastrophic‑forgetting mitigation.

Knowledge Distillationmodel efficiencyon‑policy distillation
0 likes · 15 min read
How On-Policy Distillation Cuts LLM Training Cost by 90%
360 Tech Engineering
360 Tech Engineering
Aug 7, 2026 · Artificial Intelligence

Token Compression: From Simple Text Trimming to LLM Context Governance

The article explains how token compression evolves from basic text shortening into a multi‑layered context‑governance process for large language models, balancing compression rate, semantic fidelity, constraint integrity, efficiency, stability and observability while deciding when and how to apply it.

Context ManagementLLM contextcost optimization
0 likes · 20 min read
Token Compression: From Simple Text Trimming to LLM Context Governance
ThinkingAgent
ThinkingAgent
Jun 17, 2026 · Artificial Intelligence

Why a 7B Model Can Outperform a 70B Model: The Power of Knowledge Distillation

The article explains how knowledge distillation lets a small LLM learn from a much larger teacher model, achieving near‑teacher performance while cutting inference cost, latency and memory, and provides a step‑by‑step guide, benchmark results, advanced on‑policy techniques, common pitfalls and best‑practice recommendations.

Knowledge DistillationLLMLLaMA-3
0 likes · 13 min read
Why a 7B Model Can Outperform a 70B Model: The Power of Knowledge Distillation
Top Architect
Top Architect
Feb 14, 2025 · Artificial Intelligence

DeepSeek Model Distillation: Principles, Innovations, Architecture, and Performance

This article provides an in‑depth overview of DeepSeek’s model distillation technology, covering its definition, core principles, innovative data‑model distillation integration, architecture design, training strategies, performance gains, and the challenges of scaling to multimodal data.

DeepSeekKnowledge TransferLarge Language Models
0 likes · 16 min read
DeepSeek Model Distillation: Principles, Innovations, Architecture, and Performance
Baobao Algorithm Notes
Baobao Algorithm Notes
Jun 28, 2024 · Artificial Intelligence

What Makes Gemma 2 a Competitive Open‑Source LLM? Architecture, Training, and Evaluation Insights

The article provides a detailed technical overview of Gemma 2, covering its decoder‑only transformer design, novel attention mechanisms, logit soft‑capping, RMSNorm, knowledge‑distillation training on trillions of tokens, extensive pre‑training infrastructure, and benchmark evaluations that demonstrate its competitiveness against larger proprietary models.

AIBenchmark EvaluationGemma 2
0 likes · 14 min read
What Makes Gemma 2 a Competitive Open‑Source LLM? Architecture, Training, and Evaluation Insights
AntTech
AntTech
Mar 11, 2024 · Artificial Intelligence

Can Small Language Models be Good Reasoners in Recommender Systems?

This article presents SLIM, a knowledge‑distillation framework that transfers the reasoning abilities of large language models to compact models for sequential recommendation, enhancing item representation, user profiling, and bias mitigation while achieving comparable performance with far lower computational resources.

AIEfficiencyKnowledge Distillation
0 likes · 12 min read
Can Small Language Models be Good Reasoners in Recommender Systems?
Architect
Architect
Feb 9, 2025 · Artificial Intelligence

How DeepSeek’s Model Distillation Boosts AI Efficiency and Performance

This article provides an in‑depth analysis of DeepSeek’s model distillation technology, covering its definition, core principles, innovative strategies, architecture design, training optimizations, benchmark results, efficiency gains, and the remaining challenges of applying distillation to large language models and multimodal data.

AI EfficiencyDeepSeekKnowledge Transfer
0 likes · 16 min read
How DeepSeek’s Model Distillation Boosts AI Efficiency and Performance
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 5, 2026 · Artificial Intelligence

When LLMs Invent Their Own Language: CLSR Enables Multi‑Agent Reasoning with Fewer Tokens

The ICML 2026 paper introduces CLSR, a framework that lets multiple LLM agents autonomously create compact, reusable symbolic communication protocols (LSFs), cutting generation tokens by 3–6× while preserving Chain‑of‑Thought accuracy across diverse reasoning benchmarks.

CLSRLLMlanguage symbolism
0 likes · 26 min read
When LLMs Invent Their Own Language: CLSR Enables Multi‑Agent Reasoning with Fewer Tokens
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 16, 2026 · Artificial Intelligence

Efficient Reasoning with Reward Shaping: Compressing Qwen 30B‑Series Chains by 20‑40%

The article analyzes how reward‑shaping techniques can shorten the chain‑of‑thought outputs of Qwen 30‑parameter series models by 20‑40% while preserving or slightly improving performance on AIME‑25 and out‑of‑distribution benchmarks, and it details the experimental design, strategic considerations, and practical insights behind this efficient reasoning approach.

QwenReward Shapingefficient inference
0 likes · 16 min read
Efficient Reasoning with Reward Shaping: Compressing Qwen 30B‑Series Chains by 20‑40%
PaperAgent
PaperAgent
Jul 5, 2026 · Artificial Intelligence

Uncovering the Privilege Illusion in OPD Distillation and How DOPD Solves It

The article identifies the hidden “privilege illusion” that degrades on‑policy distillation when privileged information is injected, and introduces Dual On‑policy Distillation (DOPD), a dynamic two‑stream approach that separates true ability gaps from information gaps, achieving superior performance and stability across LLM and VLM benchmarks.

DOPDLarge Language ModelsOPD
0 likes · 13 min read
Uncovering the Privilege Illusion in OPD Distillation and How DOPD Solves It
Machine Heart
Machine Heart
May 18, 2026 · Artificial Intelligence

Can Large Models Reason Deeply with Only a Few Thinking Tokens?

The paper introduces Heima, a framework that compresses chain‑of‑thought reasoning into a small set of abstract “thinking tokens” for multimodal large models, dramatically reducing generated tokens while preserving inference capability, and provides an adaptive interpreter to reconstruct human‑readable reasoning for analysis.

chain-of-thoughtefficient inferencelatent reasoning
0 likes · 12 min read
Can Large Models Reason Deeply with Only a Few Thinking Tokens?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 3, 2026 · Artificial Intelligence

Beyond Dense and MoE: JTok Module Cuts Compute by One‑Third as a New Scaling Path

The paper introduces JTok and its dynamic variant JTok‑M, a token‑indexed parameter scaling method that decouples model capacity from compute, achieving up to 35% compute reduction while delivering consistent performance gains across a wide range of downstream tasks and model sizes.

Compute EfficiencyJTokToken-indexed scaling
0 likes · 16 min read
Beyond Dense and MoE: JTok Module Cuts Compute by One‑Third as a New Scaling Path