Model Distillation

JD Tech Talk

May 22, 2025 · Artificial Intelligence

From Academic Research to Industrial Anti‑Fraud: Leveraging LLMs, Reinforcement Learning, and Model Distillation for Advertising Risk Detection

The article recounts Xiaoting’s journey from a PhD research background to leading JD.com’s ad‑fraud detection, detailing how large language models, reinforcement learning, and model distillation were applied to identify hidden address codes, reduce false‑positive rates to 0.3%, and balance accuracy with real‑time performance in a high‑traffic e‑commerce environment.

AILLMModel Distillation

0 likes · 11 min read

From Academic Research to Industrial Anti‑Fraud: Leveraging LLMs, Reinforcement Learning, and Model Distillation for Advertising Risk Detection

Architect's Guide

May 13, 2025 · Artificial Intelligence

DeepSeek Model Distillation Technology: Overview, Innovations, Architecture, Training, Performance, and Challenges

This article provides a comprehensive overview of DeepSeek's model distillation technology, detailing its definition, key innovations, architecture, training methods, performance gains, and the remaining challenges such as the implicit performance ceiling and multimodal data distillation.

AI optimizationDeepSeekModel Distillation

0 likes · 14 min read

DeepSeek Model Distillation Technology: Overview, Innovations, Architecture, Training, Performance, and Challenges

Baidu Geek Talk

Apr 9, 2025 · Artificial Intelligence

Baidu's Wenxin X1 Large Model Officially Launches on Qianfan Platform

On April 2, Baidu released its Wenxin X1 large model on the Qianfan platform, offering enterprise users and developers a multimodal, deep‑thinking AI with superior math, coding, and reasoning scores, low token‑price API access, batch inference, one‑click distillation, and rapid RAG/Agent application building.

AIAPI ServiceBaidu

0 likes · 4 min read

Baidu's Wenxin X1 Large Model Officially Launches on Qianfan Platform

DataFunTalk

Feb 28, 2025 · Artificial Intelligence

DeepSeek LLM Series (V1‑V3) and R1: Architecture, Training Strategies, Evaluation, and Distillation

An in‑depth overview of the DeepSeek LLM series (V1‑V3) and the R1 models, covering their architectures, scaling‑law experiments, data pipelines, training strategies—including MoE, MLA, FP8, multi‑step learning‑rate scheduling, reinforcement learning, and extensive evaluation results, as well as knowledge‑distillation techniques.

AI researchMixture of ExpertsModel Distillation

0 likes · 36 min read

Architect

Feb 25, 2025 · Artificial Intelligence

DeepSeek R1: Multi‑Stage Reinforcement Learning, Reward Modeling, and Distillation for a High‑Performance LLM

DeepSeek R1 builds on the DeepSeek V3 base model using a multi‑stage reinforcement learning pipeline—including GRPO optimization, rule‑based reward modeling, supervised fine‑tuning, language‑consistency rewards, rejection sampling, and distillation—to produce a high‑performing, aligned LLM capable of accurate reasoning.

DeepSeekLLM TrainingModel Distillation

0 likes · 24 min read

DeepSeek R1: Multi‑Stage Reinforcement Learning, Reward Modeling, and Distillation for a High‑Performance LLM

DataFunSummit

Feb 25, 2025 · Artificial Intelligence

Tiny‑R1‑32B‑Preview: A 5% Parameter Model Matching Deepseek‑R1‑671B Performance

On February 24, 2025, 360 and Peking University unveiled Tiny‑R1‑32B‑Preview, a medium‑scale inference model that uses only 5% of the parameters yet achieves performance comparable to the 671‑billion‑parameter Deepseek‑R1, with leading results on math, programming, and scientific benchmarks.

AI modelBenchmarkingModel Distillation

0 likes · 7 min read

Tiny‑R1‑32B‑Preview: A 5% Parameter Model Matching Deepseek‑R1‑671B Performance

Tencent Technical Engineering

Feb 21, 2025 · Artificial Intelligence

DeepSeek-R1: Enhancing Reasoning Capabilities in LLMs via Reinforcement Learning

DeepSeek‑R1 demonstrates that large‑scale reinforcement learning, especially with the novel Group Relative Policy Optimization and a rule‑based reward scheme, can markedly boost reasoning in LLMs without heavy supervised fine‑tuning, while a brief cold‑start SFT phase, two‑stage alignment, and knowledge distillation further improve performance and efficiency, despite remaining challenges such as language mixing.

Cold StartDeepSeek-R1GRPO

0 likes · 21 min read

DeepSeek-R1: Enhancing Reasoning Capabilities in LLMs via Reinforcement Learning

DataFunTalk

Feb 16, 2025 · Artificial Intelligence

Understanding Reasoning LLMs: DeepSeek R1 Variants, Inference‑Time Scaling, and Training Strategies

This article explains what reasoning language models are, outlines their strengths and weaknesses, details DeepSeek R1's three variants and their training pipelines—including pure reinforcement learning, SFT + RL, and distillation—while also discussing inference‑time scaling techniques and related research such as Sky‑T1 and TinyZero.

DeepSeekModel DistillationSupervised Fine-tuning

0 likes · 16 min read

Understanding Reasoning LLMs: DeepSeek R1 Variants, Inference‑Time Scaling, and Training Strategies

Top Architect

Feb 14, 2025 · Artificial Intelligence

DeepSeek Model Distillation: Principles, Innovations, Architecture, and Performance

This article provides an in‑depth overview of DeepSeek’s model distillation technology, covering its definition, core principles, innovative data‑model distillation integration, architecture design, training strategies, performance gains, and the challenges of scaling to multimodal data.

AI optimizationDeepSeekModel Distillation

0 likes · 16 min read

DeepSeek Model Distillation: Principles, Innovations, Architecture, and Performance

IT Architects Alliance

Feb 10, 2025 · Artificial Intelligence

DeepSeek Distillation Technology: Principles, Innovations, Performance, and Future Outlook

The article explains DeepSeek's model distillation technique, covering its fundamental knowledge‑transfer principles, unique innovations such as data‑model fusion and task‑specific strategies, impressive benchmark results, practical applications in edge and online inference, existing challenges, and future research directions.

AI optimizationEdge ComputingModel Distillation

0 likes · 15 min read

DeepSeek Distillation Technology: Principles, Innovations, Performance, and Future Outlook

Architecture Digest

Feb 7, 2025 · Artificial Intelligence

Open-Source Replication of OpenAI’s o1 Model Achieves Superior Performance with Minimal Cost

A recent study by Fei‑Fei Li’s team shows that using supervised fine‑tuning on the open‑source Qwen2.5‑32B‑Instruct model can replicate and even surpass the reasoning abilities of OpenAI’s o1‑preview at a fraction of the computational cost, demonstrating a cheap yet powerful approach to large‑language‑model development.

Model DistillationSupervised Fine-tuningbudget-forcing

0 likes · 6 min read

Open-Source Replication of OpenAI’s o1 Model Achieves Superior Performance with Minimal Cost

IT Services Circle

Feb 2, 2025 · Artificial Intelligence

OpenAI and Anthropic Accuse DeepSeek of Model Distillation and IP Infringement: Industry Reactions and Technical Overview

OpenAI and Anthropic allege that DeepSeek has illegally distilled their large language models, prompting investigations, industry satire, and a detailed look at model distillation technology, its legal implications, and the broader trends shaping AI cost, scaling laws, and market dynamics.

AI ethicsArtificial IntelligenceDeepSeek

0 likes · 10 min read

OpenAI and Anthropic Accuse DeepSeek of Model Distillation and IP Infringement: Industry Reactions and Technical Overview

DataFunSummit

Jul 10, 2024 · Artificial Intelligence

Applying Large Language Models to Recommendation Systems at Ant Group

The article presents Ant Group's research on integrating large language models into recommendation pipelines, covering background challenges, knowledge extraction, teacher‑model distillation, efficient deployment, experimental results, and future directions to improve accuracy and reduce bias.

AILLMModel Distillation

0 likes · 13 min read

Applying Large Language Models to Recommendation Systems at Ant Group

DataFunSummit

Jul 9, 2024 · Artificial Intelligence

Applying Large Language Models to Recommendation Systems at Ant Group

This article details Ant Group's research on integrating large language models into recommendation pipelines, covering background challenges, knowledge extraction, teacher‑student distillation, experimental results, and practical Q&A for improving bias, efficiency, and cold‑start performance.

AI researchAnt GroupModel Distillation

0 likes · 14 min read

Rare Earth Juejin Tech Community

May 1, 2024 · Artificial Intelligence

Hyper‑SD: Trajectory‑Segmented Consistency Model for Accelerating Diffusion Image Generation

Hyper‑SD introduces a trajectory‑segmented consistency distillation framework that combines trajectory‑preserving and trajectory‑reconstruction strategies, integrates human‑feedback learning and score distillation, and achieves state‑of‑the‑art low‑step image generation performance on both SD1.5 and SDXL models.

AI accelerationImage GenerationModel Distillation

0 likes · 10 min read

Hyper‑SD: Trajectory‑Segmented Consistency Model for Accelerating Diffusion Image Generation

Baidu Tech Salon

Nov 10, 2023 · Artificial Intelligence

Baidu Search Deep Learning Model Architecture and Optimization Practices

Baidu's Search Architecture team details how its deep‑learning models have evolved to deliver direct answer results via semantic embeddings, describes a massive online inference pipeline that rewrites queries, ranks relevance, and classifies types, and outlines optimization techniques—including data I/O, CPU/GPU balancing, pruning, quantization, and distillation—to achieve high‑throughput, low‑latency search.

BaiduGPU optimizationInference System

0 likes · 13 min read

Baidu Search Deep Learning Model Architecture and Optimization Practices

Baidu Geek Talk

Nov 9, 2023 · Artificial Intelligence

Deep Learning Model Architecture Evolution in Baidu Search

The article chronicles Baidu Search’s Model Architecture Group’s evolution of deep‑learning‑driven search, detailing the shift from inverted‑index to semantic vector indexing, the use of transformer‑based models for text and image queries, large‑scale offline/online pipelines, and extensive GPU‑centric optimizations such as pruning, quantization and distillation, all aimed at delivering precise, cost‑effective results to hundreds of millions of users.

ERNIEGPU inferenceModel Distillation

0 likes · 14 min read

Deep Learning Model Architecture Evolution in Baidu Search

DataFunTalk

Oct 2, 2023 · Artificial Intelligence

DAMO-YOLO: A High‑Efficiency, High‑Accuracy Object Detection Framework

DAMO‑YOLO is an open‑source, high‑speed and high‑precision object detection framework that leverages MAE‑NAS for low‑cost model customization, Efficient RepGFPN and HeavyNeck for enhanced multi‑scale detection, and a universal distillation technique to boost performance across model scales.

Computer VisionEfficient RepGFPNMAE-NAS

0 likes · 15 min read

DAMO-YOLO: A High‑Efficiency, High‑Accuracy Object Detection Framework

Baidu Geek Talk

Mar 16, 2023 · Artificial Intelligence

PaddleDetection v2.6 Release: PP-YOLOE Family Expansion and Advanced Detection Algorithms

PaddleDetection v2.6 expands the PP‑YOLOE family with rotating, small‑object, dense‑object, and ultra‑lightweight edge‑GPU models, upgrades PP‑Human and PP‑Vehicle toolboxes, releases semi‑supervised, few‑shot and distillation learning methods, adds numerous state‑of‑the‑art algorithms, and improves infrastructure with Python 3.10, EMA filtering and AdamW support.

BaiduComputer VisionModel Distillation

0 likes · 14 min read

PaddleDetection v2.6 Release: PP-YOLOE Family Expansion and Advanced Detection Algorithms

DataFunTalk

Jan 18, 2023 · Artificial Intelligence

Search Relevance System Architecture and Practices in QQ Browser

This article presents the QQ Browser search relevance team's experience integrating QQ Browser and Sogou search systems, detailing business overview, relevance system evolution, algorithm architecture, evaluation metrics, deep semantic matching, relevance calibration, and model distillation techniques to improve search relevance performance.

Model DistillationSearch Relevancedeep learning

0 likes · 31 min read