Tagged articles
self-distillation
25 articles · Page 1 of 1
Tsinghua researchers propose ActionOPD, an action-level online distillation method that aligns student agents with teacher decisions at the semantic action level rather than token level, enabling black-box cross-family distillation and self-distillation with improved decision-making on benchmarks like τ²-bench.
ActionOPDReinforcement Learningagent distillation
0 likes · 15 min read

This article provides a comprehensive analysis of On-Policy Distillation (OPD), covering white-box distillation with token-level probability matching, self-distillation using privileged context, black-box distillation via reward modeling, and industrial applications from MiMo, DeepSeek, and Qwen, while discussing OPD's limitations and generalization capabilities.
Black-Box DistillationGKDMiniLLM
0 likes · 56 min read

A large-scale empirical study comparing GRPO with 10 self-distillation methods across 4 tasks and 3 model sizes finds no method consistently beats GRPO; some methods show high final scores but suffer training instability, and compute-equivalent comparisons often reverse apparent gains.
GRPOReinforcement LearningReproducibility
0 likes · 10 min read

Researchers from UIUC and Tsinghua startup Astraculum built a Social World Model using a 7B LLM with continual self-distillation on 390 days of prediction market data, achieving deployment-time learning that beats static frontier models like GPT-5.6 and Claude Opus 5.
Continual LearningLLMPrediction Markets
0 likes · 11 min read

Since 2026, On‑Policy Distillation (OPD) has rapidly become a focal research area, evolving from offline teacher‑generated data to online student‑driven supervision, with advances such as OPD+, Direct OPD, weak‑to‑strong OPD, self‑distillation techniques, and preference‑optimization signals reshaping post‑training for large language models.
NLPOPDlarge language models
0 likes · 7 min read

Vision-OPD introduces an online self‑distillation framework that lets a 9B multimodal LLM internalize fine‑grained visual evidence from whole images, achieving state‑of‑the‑art results on six detailed visual‑understanding benchmarks without extra inference tools.
Vision-OPDbenchmarkfine-grained visual understanding
0 likes · 13 min read

ProLaViT introduces a progressive latent visual reasoning framework that replaces costly explicit image generation and unstable one‑shot latent inference with a structured Locate→Focus→Isolate chain, endogenous self‑distillation, and a distance‑weighted diversity loss, achieving state‑of‑the‑art accuracy on multiple visual reasoning benchmarks.
Diversity LossProLaViTlatent space
0 likes · 10 min read

InstEmb introduces learnable look‑ahead tokens and dual‑anchor alignment pooling to inject future output semantics into embeddings, enabling instruction‑following tasks such as product retrieval to achieve strong performance without any extra decoding overhead.
InstEmbLlama-3dual-anchor pooling
0 likes · 15 min read

InstEmb introduces learnable look‑ahead tokens and dual‑anchor alignment pooling to inject output‑aware semantics into instruction‑following embeddings without extra decoding, achieving strong gains on ICML‑2026 benchmarks such as FollowIR and InfoSearch while maintaining competitive performance on general sentence‑embedding tasks.
EmbeddingICML 2026Llama-3
0 likes · 15 min read

The paper presents d‑OPSD, the first on‑policy self‑distillation framework for diffusion large language models that eliminates reference solutions and extra teacher models, using only one‑tenth of RL steps while achieving equal or superior reasoning performance and markedly higher training efficiency, as demonstrated on multiple math‑reasoning benchmarks.
OPSDReinforcement Learningd-OPSD
0 likes · 7 min read

The article explains how the Chinese Ministry directives push robots from demo to operational mode, introduces LingBot‑Vision’s boundary‑centric masked modeling that embeds spatial structure into pre‑training, and shows LingBot‑Depth 2.0’s masked depth modeling dramatically improve dense depth perception on challenging surfaces while using far less data than prior SOTA models.
Boundary-Centric Masked ModelingLingBot-VisionSpace-Native Vision
0 likes · 15 min read

This article surveys the latest on‑policy distillation (OPD) research, categorizing each work as either a reinterpretation of an existing problem or a modification of a different module, and highlights the experimental findings, design choices, and trade‑offs reported across the papers.
LLMModel EfficiencyOPD
0 likes · 31 min read

This article surveys the latest research on On-Policy Distillation for large language models, covering methods that improve training stability, self‑distillation frameworks, and detailed analyses of when and why OPD succeeds or fails, with concrete experimental results and practical insights.
Entropy-AwareReinforcement Learninglarge language models
0 likes · 19 min read

The paper introduces Native Parallel Reasoner (NPR), a framework that lets language agents generate and maintain multiple reasoning paths using a three‑stage self‑distillation and parallel reinforcement‑learning training paradigm, achieving up to 4.6× speedup and significant accuracy gains across eight reasoning benchmarks.
AI ReasoningNative Parallel Reasonerbenchmark evaluation
0 likes · 18 min read

D-OPSD presents the first online self‑distillation framework for step‑distilled diffusion models, allowing them to continuously fine‑tune with only image‑text pairs, retain their fast few‑step sampling, and acquire new concepts, styles, or domain preferences without reward models.
LoRAdiffusion modelsfew-step sampling
0 likes · 10 min read

OneSearch‑V2 introduces a latent‑reasoning enhanced self‑distillation framework that augments query understanding with thought‑augmented CoT, aligns preferences via direct user behavior feedback, and achieves up to 4 % CTR lift and significant order growth without adding inference cost or latency.
LLMReinforcement Learningbehavioral feedback
0 likes · 26 min read

OneSearch‑V2 introduces a zero‑cost generative search upgrade that leverages latent‑reasoning‑enhanced self‑distillation, thought‑augmented query understanding, and behavior‑feedback preference alignment, delivering offline HitRate gains of up to 2.68 % and online CTR, buyer and order increases of roughly 4 %, 2 % and 2 % respectively.
AI rankingbehavioral feedbacke-commerce
0 likes · 24 min read

The article reviews two recent Princeton papers—SD‑ZERO, which introduces self‑revision training and on‑policy self‑distillation to turn a model’s own error traces into dense supervision, and AggAgent, which actively aggregates parallel long‑horizon trajectories—showing how internal trajectory mining can cut compute costs and boost accuracy on challenging math and code benchmarks.
AggAgentcomplex reasoninglarge language models
0 likes · 10 min read

The article explains On-Policy Distillation, derives its forward and reverse KL gradients, introduces Self‑Distillation where the policy serves as its own teacher, discusses practical implementation tricks such as extra‑knowledge injection, EMA or trust‑region teacher stabilization, and highlights benefits like reduced catastrophic forgetting, fewer Aha moments, and a narrower train‑test gap, especially for larger models.
Catastrophic ForgettingEMAKL Divergence
0 likes · 6 min read

At the start of 2026, self‑distillation dominates the most cited LLM papers, offering a teacher‑free way for large models to continually acquire new knowledge while preserving existing capabilities.
Continual LearningReinforcement Learninglarge language models
0 likes · 9 min read

This weekly roundup highlights five recent AI agent papers—PaperBanana for automated academic illustration, SDPO's self‑distillation reinforcement learning, Lumine's open‑world generalist agent, Idea2Story's pipeline for turning research ideas into narratives, and Insight Agents' fast e‑commerce insights—showcasing diverse breakthroughs in multi‑agent frameworks, self‑feedback learning, and real‑world deployment.
AI agentsReinforcement Learningautomated scientific narrative
0 likes · 8 min read


AIWalkerFeb 23, 2025 · Artificial Intelligence
D-FINE introduces Fine-grained Distribution Refinement and Global Optimal Localization Self-Distillation to overhaul DETR's bounding-box regression, achieving 54‑59% AP on COCO and Objects365 at 78‑124 FPS while surpassing YOLO and RT-DETR in both accuracy and speed.
Computer VisionDETRObject Detection
0 likes · 25 min read

This paper introduces M2SD, a dual‑branch multiple‑mixing self‑distillation framework that expands feature space, mitigates overfitting and catastrophic forgetting, and achieves state‑of‑the‑art results on CIFAR‑100, CUB‑200 and miniImageNet for few‑shot class‑incremental learning.
Few-Shot LearningM2SDclass-incremental learning
0 likes · 17 min read

The new YOLOv6 2.0 release upgrades lightweight and medium‑large models with a CSPStackRep backbone, self‑distillation, and a custom quantization pipeline, delivering up to 869 FPS for the quantized YOLOv6‑S and achieving 49.5%/52.5% AP on COCO while halving training time.
COCO benchmarkCSPStackRepObject Detection
0 likes · 6 min read


AntTechSep 9, 2022 · Artificial Intelligence
Ant Security Lab's algorithm engineer Lin Jinzheng secured two gold medals and one silver at the KDD Cup 2022, ranking first globally, by applying innovative keyword‑extraction and self‑distillation techniques to improve product search relevance and interactive risk‑control systems.
AIKDD CupProduct Search
0 likes · 4 min read
