Tagged articles
distillation
25 articles · Page 1 of 1
The article analyzes Kimi K3.1 Pro's massive 2.8T parameter MoE architecture, explains why its 1.5TB 4-bit weight exceeds any Mac's memory, and outlines three practical paths: API usage, waiting for distilled models, or running smaller MoE models locally.
APIKimi K3.1 ProMac
0 likes · 10 min read


ITPUBSep 22, 2026 · Artificial Intelligence
The article analyzes the hype around TypeSafe's Jev model, a 'System One' classifier claiming 193x speedup and 445x cost reduction, while experts like Redis creator antirez and developers Theo Browne, Simon Willison warn its narrow use cases, misleading '0% hallucination' claims, and opacity risks limit real-world utility for most developers.
AI classifierJevSystem One
0 likes · 14 min read

The article argues that reinforcement learning is crucial in post‑training because it refines and localizes chain‑of‑thought patterns learned during supervised fine‑tuning, improves model controllability, and can be complemented or substituted by distillation—especially soft distillation—to transfer high‑quality patterns from stronger teachers to weaker models.
LLMPost-Trainingchain-of-thought
0 likes · 12 min read

LiveEdit, a new streaming video editing framework from Tsinghua and HKUST, replaces bidirectional attention with causal chunk‑wise processing and a three‑stage distillation pipeline, achieving 12.66 FPS real‑time performance while preserving edit accuracy and background consistency, as demonstrated by extensive quantitative and user studies.
LiveEditReal-timecausal attention
0 likes · 9 min read

StepOPSD adds a post‑hoc, step‑aware distillation stage to multi‑turn agent reinforcement learning, splitting rollouts into controllable steps, using successful trajectories as hindsight teachers to compute token‑level advantage adjustments, and demonstrating significant gains on ALFWorld and Search‑QA tasks where reward misalignment is most severe.
ALFWorldAdvantage WeightingCredit Assignment
0 likes · 13 min read

LongCat Video Avatar 1.5 is now open‑source, delivering commercial‑grade lip sync, physical realism, long‑video stability, multi‑person interaction and 15× faster inference through Whisper‑large audio encoding, DMD 8‑step distillation and LoRA adapters, and it outperforms leading closed‑source models in extensive human‑rated benchmarks.
AIDigital HumanLongCat-Video-Avatar
0 likes · 9 min read

The article presents a two‑step cross‑architecture distillation method that replaces the quadratic softmax attention of Transformers with a learned linear attention and then maps it onto a Mamba backbone, achieving near‑teacher performance while reducing inference cost to linear time.
Cross‑ArchitectureLinear attentionMamba
0 likes · 8 min read

The article reviews three optimization paths for the Qwen3.6‑35B model—four‑bit AWQ quantization variants, the DFlash speculative decoding accelerator, and a Claude Opus‑based distillation—detailing their implementation steps, benchmark results, and guidance on selecting the best version for different hardware and performance needs.
AIDFlashQwen3.6
0 likes · 11 min read

The newly released Qwopus3.5‑v3 model combines higher‑quality reasoning chains, dedicated tool‑calling reinforcement learning, and an act‑then‑refine paradigm, delivering a 5‑point HumanEval boost, a 1.43‑point MMLU‑Pro gain, 31.7% faster inference and 24% lower token cost, while remaining runnable on a 3090 or a 16 GB MacBook, with easy deployment via GGUF, LM Studio, Ollama or llama.cpp.
Claude OpusHumanEvalMMLU-Pro
0 likes · 12 min read

This guide maps the end-to-end process for building a small AI model—from leveraging open-source base models and applying SFT with LoRA/QLoRA, through alignment techniques like DPO or ORPO, to low-cost distillation and final quantization for local deployment, while recommending free GPU resources and essential tooling.
AILoRAQLoRA
0 likes · 12 min read

The new Claude‑Opus‑4.6 distilled Qwen3.5‑v2 keeps code‑generation accuracy while cutting reasoning length by 24% and boosting per‑token correctness by 31.6%, offering a noticeable speed and cost advantage for local LLM deployment despite a 7.2% drop on MMLU‑Pro.
Claude OpusQwen3.5distillation
0 likes · 7 min read


Amap TechJan 8, 2026 · Artificial Intelligence
This article details the AI techniques behind Gaode's "Street Ranking" project, explaining the Fancy video concept, the dual training and production pipelines, and the use of SFT, reinforcement learning, MoE‑LoRA, distribution‑matching distillation, and quality‑filtering to achieve 25× faster generation with high aesthetic fidelity.
AI Video Generationdistillationmodel fine-tuning
0 likes · 24 min read

At ACL 2025, a collaborative paper introduced the Law of Capacity Gap, revealing a linear 2.5× optimal teacher‑student size relationship in language model distillation, dramatically cutting compute costs and achieving Pareto‑optimal efficiency, with the MiniMA model as a successful demonstration.
MiniMAartificial-intelligencecapacity-gap
0 likes · 7 min read

FLUX-Lightning, introduced by PaddleMIX, combines phased consistency distillation, adversarial learning, distribution‑matching distillation, and reflow loss to reduce diffusion model inference to just four steps while preserving image quality, and leverages the CINN compiler to achieve over 30% speed gains on A800 GPUs, surpassing existing SOTA acceleration methods.
AI inferenceCINNFlux
0 likes · 21 min read

This article introduces a variable‑length chain‑of‑thought distillation technique built on Alibaba Cloud PAI’s EasyDistill toolkit, presents the high‑quality OmniThought‑0528 dataset, details the training of the DistillQwen‑ThoughtY 4B/8B/32B models, and provides code and usage examples for researchers and practitioners.
DatasetLLMModel Training
0 likes · 15 min read

The Beijing University team unveils FairyR1‑32B, a 32‑billion‑parameter LLM built on DeepSeek‑R1‑Distill‑Qwen‑32B that uses self‑merging, multi‑teacher cross‑distillation, and lightweight distillation to achieve competitive math and code benchmark scores with only about 5% of the original model’s parameters.
distillationlarge language modelmodel compression
0 likes · 6 min read

This article introduces the OmniThought dataset, which annotates over two million chain‑of‑thought reasoning steps with Reasoning Verbosity and Cognitive Difficulty scores, and explains how these metrics guide the training of DistilQwen‑ThoughtX models that adapt chain length to task difficulty, achieving superior performance compared to existing distilled LLMs.
CoTDatasetLLM
0 likes · 16 min read

The article details Qwen3's three‑phase pretraining pipeline, long‑context extensions, a cold‑start long‑chain‑of‑thought dataset, reinforcement‑learning fine‑tuning with custom rewards, and a two‑stage distillation process that yields versatile, thought‑controlled language models.
PretrainingQwen3distillation
0 likes · 15 min read

Jeff Dean’s 2024 ETH Zurich talk traces fifteen years of AI breakthroughs—from the rise of neural networks and back‑propagation, through large‑scale distributed training, TPUs, Transformers, sparse MoE models, and advanced prompting techniques—showing how scaling compute, data, and clever software have driven today’s powerful Gemini models.
AIChain-of-ThoughtSparse Models
0 likes · 18 min read

Alibaba Cloud’s AI platform PAI recently saw two papers accepted at EMNLP2024—VideoCLIP‑XL, which enhances video‑text representation for long descriptions using a large video‑long‑description dataset and novel pre‑training tasks, and TAPIR, a curriculum‑planning framework that distills instruction‑following abilities of large language models—while also releasing associated models, datasets, and integration tools for users.
EMNLP2024distillationlarge-language-models
0 likes · 8 min read

Target‑Driven Distillation (TDD) is a multi‑goal distillation method that flexibly selects short‑range target steps and decouples guidance during training, enabling 4‑to‑8‑step diffusion generation that preserves high‑resolution detail, works with LoRA, ControlNet, InstantID, and outperforms existing consistency distillation techniques in speed and quality.
AI accelerationImage Generationdiffusion models
0 likes · 9 min read

This guide explains how to reduce the computational cost of large language models by preparing instruction data, optionally augmenting or refining it, deploying teacher and student models on PAI, and performing distillation training with detailed hyper‑parameter settings and sample Python scripts.
AIData AugmentationLLM
0 likes · 21 min read

OPPO details how it deploys multimodal pretrained models on resource‑constrained edge devices by compressing CLIP‑based image‑text retrieval, adapting Chinese text‑to‑image generation with LoRA and adapters, and lightweighting diffusion models through layer pruning and progressive distillation, achieving sub‑3‑second generation while preserving cloud‑level quality.
CLIPEdge DeploymentLoRA
0 likes · 18 min read

This article introduces DAMO‑YOLO, a high‑performance object detection framework that combines low‑cost model customization via MAE‑NAS, an Efficient RepGFPN with HeavyNeck for superior multi‑scale detection, and a full‑scale distillation technique, delivering faster inference, lower FLOPs, and higher accuracy across diverse industrial scenarios.
NASObject DetectionYOLO
0 likes · 15 min read

Meituan’s Search Ads team placed third in the KDD Cup 2020 Multimodalities Recall track by tackling training‑test distribution mismatch with diversified negative sampling and distillation learning, and improving text‑image matching via gated fully‑connected layers, bidirectional attention, and diversified fusion, then ensembling neural and tree models for strong NDCG gains later applied to their ad creative‑selection system.
KDD CupText‑Image Matchingdistillation
0 likes · 19 min read
