BestHub
Sign in
Tagged articles

self-distillation

21 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 3, 2026 · Artificial Intelligence

OPD Evolution: From CoT SFT to Self‑Distillation and Preference Optimization

Since 2026, On‑Policy Distillation (OPD) has rapidly become a focal research area, evolving from offline teacher‑generated data to online student‑driven supervision, with advances such as OPD+, Direct OPD, weak‑to‑strong OPD, self‑distillation techniques, and preference‑optimization signals reshaping post‑training for large language models.

Large Language ModelsNLPOPD
0 likes · 7 min read
OPD Evolution: From CoT SFT to Self‑Distillation and Preference Optimization
Machine Heart
Machine Heart
Jul 21, 2026 · Artificial Intelligence

Why Naïve Visual Reasoning Fails and How ProLaViT Enables Step‑by‑Step Latent Reasoning in Multimodal LLMs

ProLaViT introduces a progressive latent visual reasoning framework that replaces costly explicit image generation and unstable one‑shot latent inference with a structured Locate→Focus→Isolate chain, endogenous self‑distillation, and a distance‑weighted diversity loss, achieving state‑of‑the‑art accuracy on multiple visual reasoning benchmarks.

Diversity LossMultimodal LLMProLaViT
0 likes · 10 min read
Why Naïve Visual Reasoning Fails and How ProLaViT Enables Step‑by‑Step Latent Reasoning in Multimodal LLMs
JD Retail Technology
JD Retail Technology
Jul 20, 2026 · Artificial Intelligence

InstEmb: Instruction-Following Embeddings that Anticipate Future Model Outputs

InstEmb introduces learnable look‑ahead tokens and dual‑anchor alignment pooling to inject output‑aware semantics into instruction‑following embeddings without extra decoding, achieving strong gains on ICML‑2026 benchmarks such as FollowIR and InfoSearch while maintaining competitive performance on general sentence‑embedding tasks.

EmbeddingICML 2026LLaMA-3
0 likes · 15 min read
InstEmb: Instruction-Following Embeddings that Anticipate Future Model Outputs
Machine Heart
Machine Heart
Jul 9, 2026 · Artificial Intelligence

Can Your Self‑Distillation Model Do Without Reference Solutions? Introducing d‑OPSD for Diffusion LLMs

The paper presents d‑OPSD, the first on‑policy self‑distillation framework for diffusion large language models that eliminates reference solutions and extra teacher models, using only one‑tenth of RL steps while achieving equal or superior reasoning performance and markedly higher training efficiency, as demonstrated on multiple math‑reasoning benchmarks.

OPSDd-OPSDdiffusion language models
0 likes · 7 min read
Can Your Self‑Distillation Model Do Without Reference Solutions? Introducing d‑OPSD for Diffusion LLMs
Machine Heart
Machine Heart
Jul 7, 2026 · Artificial Intelligence

Space‑Native Vision: LingBot‑Vision & Depth 2.0 Advance Embodied Intelligence

The article explains how the Chinese Ministry directives push robots from demo to operational mode, introduces LingBot‑Vision’s boundary‑centric masked modeling that embeds spatial structure into pre‑training, and shows LingBot‑Depth 2.0’s masked depth modeling dramatically improve dense depth perception on challenging surfaces while using far less data than prior SOTA models.

Boundary-Centric Masked ModelingDepth EstimationEmbodied Intelligence
0 likes · 15 min read
Space‑Native Vision: LingBot‑Vision & Depth 2.0 Advance Embodied Intelligence
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 21, 2026 · Artificial Intelligence

xOPD Evolution: Mapping Recent OPD Improvements – Rephrased Same Problems vs. New Modules

This article surveys the latest on‑policy distillation (OPD) research, categorizing each work as either a reinterpretation of an existing problem or a modification of a different module, and highlights the experimental findings, design choices, and trade‑offs reported across the papers.

LLMOPDOn-Policy Distillation
0 likes · 31 min read
xOPD Evolution: Mapping Recent OPD Improvements – Rephrased Same Problems vs. New Modules
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 18, 2026 · Artificial Intelligence

From Imitation to Optimization: Recent Advances in On-Policy Distillation

This article surveys the latest research on On-Policy Distillation for large language models, covering methods that improve training stability, self‑distillation frameworks, and detailed analyses of when and why OPD succeeds or fails, with concrete experimental results and practical insights.

Entropy-AwareLarge Language ModelsOn-Policy Distillation
0 likes · 19 min read
From Imitation to Optimization: Recent Advances in On-Policy Distillation
Machine Heart
Machine Heart
May 18, 2026 · Artificial Intelligence

ICML 2026: From Single‑Threaded Thinking to Native Parallel Reasoning in Agents

The paper introduces Native Parallel Reasoner (NPR), a framework that lets language agents generate and maintain multiple reasoning paths using a three‑stage self‑distillation and parallel reinforcement‑learning training paradigm, achieving up to 4.6× speedup and significant accuracy gains across eight reasoning benchmarks.

AI reasoningLarge Language ModelsNative Parallel Reasoner
0 likes · 18 min read
ICML 2026: From Single‑Threaded Thinking to Native Parallel Reasoning in Agents
Machine Heart
Machine Heart
May 13, 2026 · Artificial Intelligence

Zero‑Cost Upgrade: OneSearch‑V2 Launches Generative Search, Boosting Buyers and Orders

OneSearch‑V2 introduces a zero‑cost generative search upgrade that leverages latent‑reasoning‑enhanced self‑distillation, thought‑augmented query understanding, and behavior‑feedback preference alignment, delivering offline HitRate gains of up to 2.68 % and online CTR, buyer and order increases of roughly 4 %, 2 % and 2 % respectively.

AI rankingbehavioral feedbacke-commerce
0 likes · 24 min read
Zero‑Cost Upgrade: OneSearch‑V2 Launches Generative Search, Boosting Buyers and Orders
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 29, 2026 · Artificial Intelligence

Dual Engine for Training and Inference: How Princeton’s SD‑ZERO and AggAgent Redefine Complex Reasoning

The article reviews two recent Princeton papers—SD‑ZERO, which introduces self‑revision training and on‑policy self‑distillation to turn a model’s own error traces into dense supervision, and AggAgent, which actively aggregates parallel long‑horizon trajectories—showing how internal trajectory mining can cut compute costs and boost accuracy on challenging math and code benchmarks.

AggAgentComplex ReasoningLarge Language Models
0 likes · 10 min read
Dual Engine for Training and Inference: How Princeton’s SD‑ZERO and AggAgent Redefine Complex Reasoning
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 22, 2026 · Artificial Intelligence

What Is On-Policy Distillation? A Deep Dive into On-Policy and Self-Distillation

The article explains On-Policy Distillation, derives its forward and reverse KL gradients, introduces Self‑Distillation where the policy serves as its own teacher, discusses practical implementation tricks such as extra‑knowledge injection, EMA or trust‑region teacher stabilization, and highlights benefits like reduced catastrophic forgetting, fewer Aha moments, and a narrower train‑test gap, especially for larger models.

EMAKL DivergenceOn-Policy Distillation
0 likes · 6 min read
What Is On-Policy Distillation? A Deep Dive into On-Policy and Self-Distillation
HyperAI Super Neural
HyperAI Super Neural
Feb 6, 2026 · Artificial Intelligence

Latest Advances in AI Agents: PaperBanana, SDPO, Lumine, Idea2Story, and Insight Agents

This weekly roundup highlights five recent AI agent papers—PaperBanana for automated academic illustration, SDPO's self‑distillation reinforcement learning, Lumine's open‑world generalist agent, Idea2Story's pipeline for turning research ideas into narratives, and Insight Agents' fast e‑commerce insights—showcasing diverse breakthroughs in multi‑agent frameworks, self‑feedback learning, and real‑world deployment.

AI agentsautomated scientific narrativemulti-agent systems
0 likes · 8 min read
Latest Advances in AI Agents: PaperBanana, SDPO, Lumine, Idea2Story, and Insight Agents
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Mar 19, 2024 · Artificial Intelligence

M2SD: Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning

This paper introduces M2SD, a dual‑branch multiple‑mixing self‑distillation framework that expands feature space, mitigates overfitting and catastrophic forgetting, and achieves state‑of‑the‑art results on CIFAR‑100, CUB‑200 and miniImageNet for few‑shot class‑incremental learning.

M2SDclass-incremental learningfeature augmentation
0 likes · 17 min read
M2SD: Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning
AntTech
AntTech
Sep 9, 2022 · Artificial Intelligence

Ant Security Lab Wins Two Golds and One Silver at KDD Cup 2022 with Advanced Keyword Extraction and Self‑Distillation for Product Search

Ant Security Lab's algorithm engineer Lin Jinzheng secured two gold medals and one silver at the KDD Cup 2022, ranking first globally, by applying innovative keyword‑extraction and self‑distillation techniques to improve product search relevance and interactive risk‑control systems.

AIKDD CupProduct Search
0 likes · 4 min read
Ant Security Lab Wins Two Golds and One Silver at KDD Cup 2022 with Advanced Keyword Extraction and Self‑Distillation for Product Search