This article explains model distillation through a question rewriting example, detailing how teacher models provide demonstrations to train smaller student models, the differences between distillation, fine-tuning, and quantization, and practical pitfalls like data quality and student capacity limits.
Jev, a new System One model from TypeSafe AI, outputs calibrated decisions instead of tokens using RLCD rather than RLHF, offering speed, generalization, and calibrated confidence that could shrink business post-training roles for classification, routing, and filtering tasks, leaving only constrained long-text generation and non-standard schema extraction.
This article provides a comprehensive overview of DeepSeek's model distillation technology, detailing its definition, key innovations, architecture, training methods, performance gains, and the remaining challenges such as the implicit performance ceiling and multimodal data distillation.
This article analyzes TokenRouter, a NeurIPS 2026 system that enables efficient token-level LLM routing through a request-centric programming interface, asynchronous model handoffs with state preservation, and throughput-optimized delayed batching, demonstrating 2-64x throughput gains over baselines across multiple routing algorithms and workloads.
Token Superposition Training (TST) speeds up large‑language‑model pre‑training by up to 2.5× without altering model architecture or compute budget, using a superposition phase that averages token embeddings into bags and predicts groups of tokens, followed by a standard recovery phase, as demonstrated on 10B‑parameter MoE and smaller models.
TwigVLM introduces a lightweight “twig” module that prunes visual tokens early and enables self‑speculative decoding, achieving up to 154% speedup on long‑text generation while preserving 96% of original LVLM accuracy, as demonstrated on LLaVA‑1.5‑7B and other benchmarks.
This article surveys the latest research on On-Policy Distillation for large language models, covering methods that improve training stability, self‑distillation frameworks, and detailed analyses of when and why OPD succeeds or fails, with concrete experimental results and practical insights.
This article reviews ten recent papers that tackle the over‑thinking problem in large language models by shortening chain‑of‑thought reasoning, introducing dynamic early‑exit, adaptive thinking triggers, and reinforcement‑learning‑based training, showing how models can maintain or improve accuracy while dramatically reducing token usage and latency.
This article explains how Alibaba Cloud's AI platform PAI leverages the EasyDistill framework for post‑training model optimization, covering knowledge distillation concepts, data synthesis techniques, basic and advanced distillation training, the DistilQwen model family, real‑world customer cases, and step‑by‑step practical demos.
This curated reading list compiles essential papers on AI agents, task planning, hallucination mitigation, multimodal models, image/video generation, foundational LLM research, open‑source large models, fine‑tuning techniques, and performance optimization, providing a comprehensive roadmap for anyone aiming to master modern generative AI.
On‑Policy Distillation (OPD) is widely used for post‑training large language models, but the sampled‑token variant often becomes unstable due to token‑level reward imbalance, teacher‑student signal mismatch on student‑generated prefixes, and tokenizer mismatches; this article analyses the bias‑variance trade‑off, identifies three root failure modes, and proposes a teacher‑top‑K local‑support‑set objective with top‑p rollout and special‑token masking that yields more stable training and better performance on both math and agentic benchmarks.
HOMURA introduces a reinforcement learning framework with KL regularization and dynamic syllable-ratio rewards to achieve precise length control in LLM translation for subtitles and dubbing, outperforming baselines on the Sand-Glass benchmark and deployed at scale on Bilibili.
This article provides an in‑depth technical analysis of DeepSeek’s model distillation technology, covering its core principles, innovative data‑model fusion strategies, architecture design, training optimizations, performance benchmarks, and the remaining challenges of scaling distillation to multimodal tasks.
Yuxian Gu, a 2021 Tsinghua PhD and 2025 Special Scholarship laureate, has joined DeepSeek, bringing expertise in pre‑training data selection, knowledge‑distillation for model compression, and efficient model architectures such as Jet‑Nemotron, which outperforms leading open‑source LLMs with up to 53.6× speedup on H100.
The article analyzes the prefix‑failure problem in on‑policy distillation, introduces Relay‑OPD with a handoff trigger that lets a teacher model intervene locally, and shows through eight math‑reasoning benchmarks that this approach improves accuracy by up to 7.3% while cutting training trajectory length by more than half.
The LightReasoner paper from Hong Kong University shows that small language models can guide large models on critical reasoning steps, achieving up to 90% faster inference and significant accuracy gains across multiple math benchmarks.
This guide walks through setting up a Kunlun P800 XPU host, preparing Docker containers, deploying Qwen3‑8B/‑32B/‑VL models with vLLM‑Kunlun, benchmarking performance, and running full‑parameter DPO training using LLaMA‑Factory, providing scripts, configuration files, and troubleshooting tips for AI engineers.
This article critically examines R1‑Zero‑style training by analyzing foundation models and reinforcement learning, uncovering pre‑training and optimization biases, proposing an unbiased Dr. GRPO method, and demonstrating a minimalist 7B‑model recipe that achieves new state‑of‑the‑art performance on AIME 2024.
This article presents the PLUG 270‑billion‑parameter Chinese language model and introduces a parameter‑efficient sparsity training (PST) framework that combines unstructured and structured pruning with low‑rank decomposition to dramatically reduce model size while preserving downstream performance.
The article explains how overly long prompts dilute a large‑model’s attention, causing slower responses and contradictory outputs, and introduces a three‑step prompt‑decay audit—density measurement, slimming, and versioned output—that cuts response time from 1.8 s to 0.6 s, triples logical density, and reduces hallucinations by 60 %.
