Meta's World-T³ introduces test-time training as a fixed-size, camera-pose-indexed memory that writes historical observations into fast weights, enabling long-horizon consistent video generation at 20.6 FPS on a single GPU while outperforming baselines on revisit consistency metrics.
This article provides a comprehensive analysis of On-Policy Distillation (OPD), covering white-box distillation with token-level probability matching, self-distillation using privileged context, black-box distillation via reward modeling, and industrial applications from MiMo, DeepSeek, and Qwen, while discussing OPD's limitations and generalization capabilities.
Google researchers identify two numerical bugs in the standard softmax-cross-entropy implementation used for two decades: Softmax Collapse zeros out gradients beyond a precision cliff, and Softmax Cancellation creates a noise floor in distillation; fixes using stable log-space computations eliminate Slingshot loss spikes in grokking and improve convergence across precisions.
Yan Wang's TraceForge method generates synthetic chain-of-thought traces via adversarial training on family-model data, legally outperforming direct distillation of closed-source models like Claude and GPT on coding benchmarks without using any proprietary tokens.
Princeton researchers introduce QUEEN, a 4B-parameter model that combines a strong chess engine with a language model via gated cross-attention bridges, trained through a four-stage QA curriculum and iterative search distillation to achieve 2697 Elo while generating natural-language move explanations.
Researchers from Peking University, Tsinghua University, and Alibaba introduce SparkDiffusion, a unified framework combining sparse attention, few-step distillation, and FP8 quantization to achieve 265× acceleration for DiT video generation on a single RTX 5090, overcoming the high-sparsity trap that degrades quality at extreme sparsity.
This article analyzes five fundamental shifts in LLM post-training over the past six months: moving from monolithic RL to expert distillation (MOPD), online distillation as a 10x cheaper RL alternative, refined RLVR techniques addressing entropy collapse and exploration, SFT-RL distribution alignment, and data quality as an irrecoverable hard constraint.
This article analyzes five major paradigm shifts in large model post-training over the past six months, covering expert distillation, online distillation as RL alternative, RLVR refinements, SFT-RL distribution mismatch, and data quality as irreducible constraint, with specific papers and metrics.
Tsinghua-led research shows On-Policy Distillation (OPD) achieves over 70% of full-data performance with just one training example, revealing that data coverage saturates quickly while algorithmic absorption speed becomes the bottleneck.
This article explains model distillation through a question rewriting example, detailing how teacher models provide demonstrations to train smaller student models, the differences between distillation, fine-tuning, and quantization, and practical pitfalls like data quality and student capacity limits.
This article explains knowledge distillation for large language models, comparing compression techniques, detailing target and feature distillation mechanisms, showcasing DeepSeek's distillation of 671B models into smaller Qwen and LLaMA variants, and providing two practical implementation paths: instruction distillation via API and classic logits-based PyTorch code with training tips.
This article explains why and how knowledge distillation lets a 1.5 B parameter model surpass much larger LLMs on math benchmarks, detailing the underlying soft‑label transfer, temperature tuning, various distillation families, engineering pipelines, and the practical trade‑offs that bound its success.
AI EngineeringMath Reasoningknowledge distillation
The article explains why large language model (LLM) inference is prohibitively expensive, outlines the three main drawbacks—high API cost, latency, and hardware requirements—and shows how knowledge distillation can reduce these costs by up to 500×, providing a detailed 7‑step workflow, white‑box vs black‑box methods, code examples, and compliance considerations.
The paper presents d‑OPSD, the first on‑policy self‑distillation framework for diffusion large language models that eliminates reference solutions and extra teacher models, using only one‑tenth of RL steps while achieving equal or superior reasoning performance and markedly higher training efficiency, as demonstrated on multiple math‑reasoning benchmarks.
The DOPD paper introduces an advantage‑aware dual distillation framework that eliminates the privilege illusion, dynamically selects token‑wise strategies, and delivers up to 7.5‑point gains on LLM benchmarks while closing 89.8% of the teacher‑student gap and showing strong robustness across model sizes.
Yuxian Gu, a 2021 Tsinghua PhD and 2025 Special Scholarship laureate, has joined DeepSeek, bringing expertise in pre‑training data selection, knowledge‑distillation for model compression, and efficient model architectures such as Jet‑Nemotron, which outperforms leading open‑source LLMs with up to 53.6× speedup on H100.
DeepSeekEfficient Model Architectureartificial intelligence
MCompassRAG introduces a semantic‑compass approach that attaches topic metadata to coarse chunks, eliminating the need for fine‑grained splitting, reranking, or LLM calls during inference, and achieves an average 8.24% information‑efficiency gain and over five‑fold latency reduction across six complex retrieval benchmarks.
The article argues that software engineering has spent the past fifty years merely managing human uncertainty rather than true engineering, and that large language models now make it possible to replace low‑level cognition with energy‑driven intelligence, demanding a shift to an AI‑centered paradigm, closed‑loop automation, and a new focus on scenario‑driven knowledge distillation.
The article explains how knowledge distillation lets a small LLM learn from a much larger teacher model, achieving near‑teacher performance while cutting inference cost, latency and memory, and provides a step‑by‑step guide, benchmark results, advanced on‑policy techniques, common pitfalls and best‑practice recommendations.
A recent Nature paper reveals that large language models can inherit hidden behavioral preferences from teacher models through subconscious learning, even when training data lack explicit semantic signals, leading to significant misalignment risks demonstrated across numeric, code, and chain‑of‑thought experiments.
The article recounts how the Postal Savings Bank of China’s software R&D team built a domain‑specific corpus, applied a three‑stage BERT training pipeline and knowledge‑distillation to lift transaction‑type classification accuracy above 95%, delivering finer customer profiling, scenario‑based marketing calendars and a roadmap for future data‑driven banking.
The article explains how On-Policy Distillation (OPD) combines on‑policy sampling with dense teacher feedback via reverse KL to address low signal density, distribution shift, and capability interference in large‑model post‑training, and compares implementations by Qwen3, GLM‑5, MiMo‑V2 and DeepSeek‑V4.
The article examines the bottleneck of synchronous rollout in large‑model post‑training, proposes an asynchronous design using gradient accumulation and a global micro‑batch count to preserve loss equivalence, and introduces LogitsExpress for efficient top‑K knowledge‑distillation communication, all implemented in the lightweight LiteScale framework.
Post-Trainingasynchronous rolloutdistributed training
EUPE introduces a three‑stage “scale‑then‑shrink” distillation pipeline that first trains a large proxy model to absorb heterogeneous expert knowledge and then compresses it into an 86M encoder, achieving state‑of‑the‑art performance on image classification, dense prediction and vision‑language tasks on an iPhone with only 62 ms latency.
A large-scale benchmark of 20 pretrained ViT teachers across 11 families shows that attention copy and distillation improve some models but hurt others—especially DINOv2, CLIP, and BEiTv2—due to architecture mismatches, and adding the teachers' native components to students restores the lost performance.
APO (Autonomous Preference Optimization) converts the drift and conflict among multiple teacher multimodal LLMs into dynamic negative constraints while treating consensus as a positive preference, enabling robust concept alignment and superior diagnostic accuracy on the CXR‑MAX benchmark, as demonstrated by extensive ICML‑2026 experiments.
The paper introduces APO, a novel autonomous preference optimization framework that converts concept drift among multiple teacher multimodal LLMs into dynamic negative constraints and treats consensus as a positive preference, achieving robust concept alignment and surpassing strong teachers on a high‑risk medical X‑ray benchmark.
Deploying large AI models to tiny embedded devices like Arduino and Raspberry Pi requires aggressive model slimming through quantization, pruning, and distillation, careful selection of runtimes such as TensorFlow Lite, and addressing power, latency, and debugging challenges to achieve real‑time inference.
This article details an end-to-end geolocation search solution that unifies pre‑training, recall, and ranking with shared representations, knowledge distillation, and S2 spatial encoding to handle massive POI data, multi‑factor relevance, and real‑time response constraints.
The article explains why pretrained LLMs cannot call tools, then breaks down the three‑stage training pipeline—Supervised Fine‑Tuning, Reinforcement Learning from Human Feedback, and knowledge distillation—showing how each step teaches models to read tool schemas, decide when to invoke a tool, generate JSON calls, and finally transfer the capability to smaller models with LoRA.
This article walks you through a complete knowledge‑distillation workflow that turns Elon Musk’s decision‑making logic into a reusable AI skill, covering source collection, Obsidian setup, a six‑step prompting chain, adding personal commentary, and packaging the result for manual or automated AI use.
The article explains why generic AI is insufficient, defines a Skill as the minimal unit of specialized AI, and details a three‑layer knowledge‑distillation methodology—knowledge, logic, style—to build practical person‑ and book‑based AI Skills, illustrated with a complete Wang Yangming Skill implementation and common pitfalls.
The article examines the hype around AI tools that turn employees' chats, documents, and emails into a compact “.skill” file, arguing that this so‑called “distillation” is merely knowledge capture, while true value—judgment, responsibility, and intuition—remains uncapturable and the anxiety surrounding it is deliberately manufactured.
The article reviews three open‑source GitHub projects—nuwa.skill, yourself‑skill, and anti‑distill—that aim to distill a person’s thinking into AI‑driven digital twins, explore practical examples, and discuss how such tools can preserve personal knowledge while preventing corporate exploitation.
The article examines how open‑source projects that distill a colleague’s knowledge into AI‑driven “skill” files raise profound concerns about talent displacement, loss of apprenticeship pathways, and the unchecked power of automated decision‑making in the tech industry.
Microsoft’s Online Experiential Learning framework lets large language models continuously self‑evolve after deployment by extracting experience from user interactions and consolidating it into model parameters, eliminating the need for human labels, reward models, or server‑side environment access, and demonstrating scalable gains across tasks and model sizes.
The article examines two open‑source Claude Code projects—colleague‑skill, which captures a departing teammate's knowledge into an AI‑driven Skill, and anti‑distill, which sanitizes that Skill to protect personal expertise—detailing their motivations, implementation, and community reaction.
Nvidia’s latest aggregated vision foundation model consolidates detection, segmentation, and other visual tasks into one network, eliminating the complexity and resource waste of multi‑model stacks; the article explains the challenges of resolution balance and teacher distribution, outlines three model generations (RADIOv2.5, C‑RADIOv3, C‑RADIOv4), and details the novel multi‑teacher distillation techniques that boost performance across benchmarks.
The paper introduces Ψ‑NN, a knowledge‑distillation framework that automatically discovers physics‑consistent network structures for PINNs, eliminating the need for manually imposed loss‑function constraints and achieving faster convergence, higher accuracy, and transferable architectures across PDE problems.
NanoSD distills Stable Diffusion 1.5 into a 130 M‑parameter model that runs inference in 20 ms on a Qualcomm SM8750 NPU, using hardware‑aware module pruning, module‑level knowledge distillation, and Bayesian optimization to achieve Pareto‑optimal quality‑efficiency trade‑offs for on‑device image restoration.
On-Policy Context Distillation (OPCD) compresses transient in‑context knowledge into LLM parameters, allowing models to permanently retain problem‑solving experience without ground‑truth labels; the article details the OPCD framework, training steps, teacher‑student configurations, and experimental results on math, games, and system‑prompt tasks, highlighting its advantages over traditional context distillation.
The paper introduces T‑LLM, a time‑distillation framework that transfers predictive behavior from a lightweight teacher model to a general‑purpose LLM, enabling accurate multivariate time‑series forecasting across full‑sample, few‑shot, and zero‑shot settings while eliminating the need for large‑scale pre‑training.
This article details Ximalaya's journey from traditional multi‑stage recommendation pipelines to generative AI‑driven models, covering business challenges, architectural and model differences, phased deployments, knowledge distillation, semantic ID encoding, decoder‑only strategies, extensive offline and online evaluations, and future research directions.
The article analyzes the Qwen3‑VL‑Embedding and Qwen3‑VL‑Reranker models, detailing their unified vector space, multi‑stage training pipeline, Matryoshka representation learning, quantization techniques, massive synthetic data generation, and benchmark results that push multimodal retrieval performance to a new state‑of‑the‑art.
Resemble AI’s open‑source Chatterbox‑Turbo reduces TTS generation from ten steps to one, enabling high‑sample‑rate, lossless voice cloning from a 5‑10 second reference while supporting emotional control, side‑language tags, and embedded watermarking for real‑time applications across chatbots, games, podcasts, and education.
This paper introduces SeDi, a semantics‑ and distribution‑aware cross‑tokenizer knowledge distillation framework that aligns teacher and student token spaces via bipartite graph components and top‑K re‑encoding, achieving state‑of‑the‑art performance and lower exposure bias on multiple LLM benchmarks.
AI researchSemantic Alignmentcross-tokenizer distillation
This article explains the TLM large‑model development platform and details how knowledge distillation—using soft labels, temperature scaling, and combined loss functions—compresses teacher models into efficient student models, with practical steps and evaluation on the platform.
This paper presents a collaborative framework where a large language model generates high‑quality synthetic samples to augment a lightweight model, dramatically improving few‑shot user‑complaint intent recognition in 5G networks, achieving a 21% boost for rare categories and a 9% overall accuracy gain.
Data AugmentationFew-Shot Learningcomplaint intent detection
EMNLP 2025 in Suzhou will feature Alibaba Cloud’s AI platform PAI presenting four accepted papers on knowledge distillation, small‑model reasoning, distilled reasoning models, and an automated RAG benchmark framework, alongside exhibition demos, networking events, and recruitment opportunities for AI talent.
AI PlatformEMNLP 2025Retrieval-Augmented Generation
Thinking Machines Lab introduces On-Policy Distillation, a post‑training technique that matches reinforcement‑learning performance while reducing compute cost by up to tenfold, and demonstrates its effectiveness through extensive experiments on reasoning, personalization, and catastrophic‑forgetting mitigation.
Model EfficiencyOn-Policy DistillationReinforcement Learning
This article summarizes four recent arXiv papers on time‑series forecasting, covering a hierarchical knowledge‑distillation framework for extreme events, a graph‑enhanced multimodal fusion network, an interpretable unsupervised anomaly detector, and an adaptive masking loss that improves prediction accuracy.
The article argues that combining large, high‑capacity models with lightweight, fine‑tuned small models can cut costs, lower latency, enable specialized vertical tasks, and shift development from chasing ever‑bigger models toward optimal system architectures, outlining key techniques such as state‑space models, knowledge distillation, and staged fine‑tuning.
This article explains the principles, key methods, and practical effects of model quantization, pruning, and knowledge distillation, comparing their advantages and disadvantages, and showing how combining these techniques enables compact, high‑performance AI models on resource‑constrained devices.
This article introduces Pos2Distill, a novel knowledge‑distillation framework that transfers capabilities from advantageous to disadvantaged positions in large language models, effectively mitigating positional bias and improving performance on long‑text retrieval and in‑context reasoning tasks.
in-context reasoningknowledge distillationlarge language models
Large language models often generate confident but false statements—a phenomenon called hallucination—because they predict the next token based on statistical patterns rather than factual understanding, and this article explains the underlying mechanisms and practical mitigation strategies.
This article explains how Alibaba Cloud's AI platform PAI leverages the EasyDistill framework for post‑training model optimization, covering knowledge distillation concepts, data synthesis techniques, basic and advanced distillation training, the DistilQwen model family, real‑world customer cases, and step‑by‑step practical demos.
The article introduces KAT‑V1 AutoThink, a dual‑mode large language model that automatically switches between thinking and non‑thinking modes based on problem difficulty, details its novel training paradigm, reinforcement‑learning enhancements, performance benchmarks against leading open‑source models, and provides open‑source resources for further research.
The 20th ICCV conference announced 8 papers from Tencent Youtu Lab covering stylized face recognition, AI‑generated image detection, heterogeneous knowledge distillation, multi‑conditional diffusion, multimodal LLM distillation, palmprint recognition, low‑light vision, and oracle bone script decipherment, each pushing the frontier of computer vision and AI research.
EasyDistill, an open-source framework from Alibaba Cloud PAI, streamlines knowledge distillation for large language models, introducing the DistilQwen-ThoughtX series with variable-length chain-of-thought reasoning, and provides comprehensive best-practice guidance for training, fine-tuning, evaluation, compression, and deployment via the PAI-ModelGallery.
DeepKD introduces a double‑layer decoupling framework and a dynamic top‑K mask that adaptively denoises low‑confidence logits, addressing conflicts between target and non‑target knowledge flows; extensive experiments on CIFAR‑100, ImageNet‑1K, and MS‑COCO demonstrate consistent accuracy gains and state‑of‑the‑art performance.
The DeepSeek‑R1‑0528 update brings major benchmark gains, reduced hallucinations, JSON output and function‑calling support, enabling seamless integration with the Model Context Protocol (MCP) for flexible tool use, while a new 8B distilled model demonstrates knowledge‑distillation benefits for resource‑constrained deployments.
EasyDistill, an open‑source toolkit from Alibaba Cloud AI Platform, streamlines knowledge distillation of large language models by offering modular data synthesis, black‑box and white‑box training, reinforcement‑learning and preference‑optimization techniques, enabling the creation of compact, high‑performance DistilQwen models and accompanying datasets.
The paper introduces an explainable LLM framework (ELLM‑rele) that uses chain‑of‑thought reasoning and a multi‑dimensional knowledge distillation pipeline to compress large‑model relevance judgments into lightweight student models, achieving superior offline relevance scores and online click‑through and conversion improvements in Taobao’s search advertising.
The article introduces the DistilQwen2.5‑R1 series, which leverages a novel knowledge‑distillation pipeline—including CoT data evaluation, improvement, and validation—to transfer deep reasoning abilities from large models like DeepSeek‑R1 to compact models, achieving superior performance across math, code, and scientific benchmarks and providing open‑source checkpoints and deployment guides for practical use.
AI inferencebenchmark evaluationknowledge distillation
The article explains how knowledge distillation transfers capabilities from large to smaller diffusion models, covering hard and soft labels, temperature scaling, and contrasting it with data distillation, while detailing techniques such as consistency models, progressive distillation, adversarial distillation, and adversarial post‑training for model compression and step reduction.
The article analyzes why larger teacher models can hurt student performance in autoregressive language model distillation, reveals that different tokens require distinct teaching modes, proposes an Adaptive Token‑wise Knowledge Distillation (ATKD) method, and shows through extensive experiments that ATKD consistently improves accuracy by about 3 % and enhances generalization across model sizes.
adaptive teachingautoregressive language modelsknowledge distillation
The article provides an in‑depth overview of DeepSeek’s large language model, detailing its mixture‑of‑experts and Transformer foundations, novel attention mechanisms, load‑balancing, multi‑token prediction, FP8 mixed‑precision training, and various training regimes such as knowledge distillation and reinforcement learning.
This article introduces DistilQwen2.5, a lightweight LLM series built on Qwen2.5 that uses a novel two‑layer distillation framework, instruction‑data optimization, and parameter‑fusion techniques to achieve higher performance while drastically reducing computational cost and deployment overhead.
DeepSeek’s distillation technology combines data and model distillation to transfer knowledge from large teacher models to compact student models, detailing its definitions, principles, key innovations, architecture, training methods, performance gains, and challenges, especially in multimodal contexts.
This paper proposes a multi‑teacher distillation framework for continual learning that combines active data rehearsal with feature‑decoupled distillation, demonstrating superior performance on PASCAL VOC and COCO benchmarks while mitigating catastrophic forgetting and balancing stability‑plasticity trade‑offs.
Knowledge Distillation transfers the soft output probabilities of a high‑capacity teacher network to a compact student model, enabling significant reductions in parameters and compute while preserving accuracy, and is widely used for model compression, inference acceleration, and transfer learning across resource‑constrained devices.
deep learningknowledge distillationmodel compression
This article explores DeepSeek’s cutting‑edge distillation technology, detailing its definition, underlying principles, innovative data‑model fusion, architecture choices, training strategies, performance gains over large language models, and the remaining challenges in knowledge transfer and multimodal data processing.
This article explains DeepSeek's knowledge‑distillation approach for compressing large language models into small, efficient student models, details step‑by‑step local deployment requirements, performance optimizations, and highlights the cost, privacy, and application benefits of running the distilled model on‑premise.
This article introduces the DeepSeek family of models, explains the concepts of online search and deep reasoning, details the two‑phase training pipeline with data augmentation and reinforcement learning, and showcases practical experiments and deployment examples for the R1 and distilled variants.
This article demystifies DeepSeek R1 by explaining key concepts such as online search integration and the R1 model, detailing its two‑phase training pipeline, core techniques like iterative data enhancement, and showcases practical reproductions, benchmark tests, and deployment examples for AI developers.
The article examines DeepSeek‑V3’s low‑cost training using 2048 H800 GPUs, explains how knowledge distillation and high‑quality data improve efficiency, discusses expert concerns about training on AI‑generated content, and outlines the limitations and ceiling effect of distillation techniques.
This article explains knowledge distillation—a technique introduced by Geoffrey Hinton that transfers knowledge from large teacher models to compact student models—covering its core concepts, loss functions, various distillation strategies, notable applications in edge computing, federated learning, continual learning, and emerging research directions.
This article explains how knowledge distillation enables massive AI models to be compressed and deployed efficiently, covering its principles, classification dimensions, implementation steps, innovative practices at DeepSeek, real‑world applications, and future research directions.
This article explains knowledge distillation as used in DeepSeek's R1 model, detailing how a large teacher model transfers knowledge to a smaller student model via soft targets and loss functions like KL divergence, enabling efficient deployment on resource-constrained devices.
Since the launch of ChatGPT, 58.com has built a Model‑as‑a‑Service platform called LingXi that trains and serves domain‑specific large language models, supports over a hundred internal scenarios with daily inference exceeding ten million calls, and continuously improves performance through quantization, GPU optimization, model miniaturization, and advanced AI applications such as interview assistants, voice agents, and RAG‑enabled agents.
KwaiCoder-23BA4-v1 is a 23B wide MoE code‑completion model that achieves state‑of‑the‑art performance on HumanEval, BigCodeBench and Fill‑in‑Middle benchmarks by using high‑quality data, a cost‑effective training pipeline that combines model pruning, knowledge distillation and fine‑grained merging, and extensive ablation studies.
SnapGen is a 379 M‑parameter text‑to‑image diffusion model that produces 1024 px images on mobile devices in about 1.4 seconds, using a compact U‑Net design, multi‑stage knowledge distillation, step distillation, and optimized training tricks to outperform much larger models on standard benchmarks.
SnapGen introduces a compact 379M‑parameter diffusion model that produces 1024‑pixel text‑to‑image results in about 1.4 seconds on a mobile device, achieving competitive FID scores and outperforming much larger models through a series of architecture refinements, advanced training tricks, and multi‑level knowledge distillation.
This paper presents SiCLIP, a framework that simplifies the Transformer architecture, combines weight‑sharing, multi‑stage knowledge distillation, and a novel pair‑matching loss with synthetic captions to train a competitive CLIP model using only one RTX3090 GPU and 1 TB of storage, achieving state‑of‑the‑art data‑size‑parameter‑accuracy trade‑offs.
This article walks through the various loss functions used in OpenRLHF—including GPTLMLoss, KDLoss, DPOLoss, KTOLoss, and reward model losses—explaining their mathematical foundations, implementation details, and practical considerations for RLHF training.
The paper introduces TAPIR, a task‑aware curriculum planning framework that distills instruction‑following abilities from black‑box LLM teachers into smaller student models by filtering difficult prompts, resampling tasks, enhancing response styles, and iteratively optimizing across multiple training rounds, achieving superior performance on benchmark evaluations.
This article introduces DistilQwen2, a lightweight language model derived from Qwen2 via knowledge distillation, detailing its data collection, instruction‑data optimization, training strategies, extensive benchmark evaluations, and practical deployment guides for developers and enterprises.
This article presents TransLLM, a cross‑language migration framework that enables high‑quality conversational LLMs to be transferred to low‑resource languages by preserving advanced capabilities through Recovery KD, LoRA‑based continual pre‑training, and a translation‑thinking‑chain, with extensive experiments showing superior performance and safety over ChatGPT and GPT‑4.
On June 27, 2024, Xiaohongshu’s technical team will livestream a two‑hour session across WeChat Channels, Bilibili, Douyin and Xiaohongshu, showcasing six top‑conference papers on large‑model advances—including early‑stopping and fine‑grained self‑consistency, novel evaluation methods, negative‑sample‑assisted distillation, and LLM‑based note recommendation—followed by a Q&A and recruitment briefing.
This article presents SLIM, a knowledge‑distillation framework that transfers the reasoning abilities of large language models to compact models for sequential recommendation, enhancing item representation, user profiling, and bias mitigation while achieving comparable performance with far lower computational resources.
This article reviews recent advances that enhance dual‑tower recommendation models by injecting interaction information through various knowledge‑distillation strategies and interaction‑enhanced architectures, summarizing methods such as PFD, ENDX, TRMD, VIRT, Distilled‑DualEncoder, ERNIE‑Search, ColBert, IntTower and MVKE.
Metis combines deterministic finite automata conversion, byte‑level RNN training, and knowledge‑distilled random‑forest models to replace traditional regex matching on resource‑constrained network devices, delivering comparable accuracy while achieving up to 74× higher throughput and significant resource savings in DDoS protection and P4 forwarding.
The article introduces Metis, an AI‑based solution that replaces traditional regular‑expression matching for network traffic inspection, offering faster, more accurate detection, a compact model deployable on resource‑constrained P4 switches, and significant performance and cost benefits for cloud gateway security.
The AAAI‑2024 paper introduces a Negative Sample Assisted Distillation framework—comprising Negative Assistance Training, Negative Calibration Enhancement, and Adaptive Self‑Consistency—that leverages both correct and incorrect reasoning examples to train a compact LLaMA‑7B student, achieving up to 75.75 % accuracy gains over fine‑tuning on MATH and improving out‑of‑domain benchmarks.
This article systematically reviews the coarse‑ranking stage of recommendation systems, comparing it with recall and fine‑ranking, defining evaluation metrics, detailing sample design, presenting two technical routes, and exploring optimization directions such as dual‑tower models, knowledge distillation, lightweight fully‑connected layers, multi‑objective and multi‑scenario modeling, followed by practical case studies and results.
The article surveys Baidu Search’s intelligent question‑answering system, tracing its evolution from feature‑engineered retrieval to large pre‑trained and generative models, and detailing hierarchical readers, multi‑teacher distillation, retrieval‑enhanced generation, and instruction decomposition as key techniques for delivering fast, accurate, citation‑rich answers.
This article reviews the evolution of machine question answering, from early feature‑engineered systems to modern large‑language‑model‑driven retrieval‑augmented generation, outlines Baidu Search’s current Retriever‑Reader architecture, discusses challenges such as semantic complexity, latency and answer quality, and presents solutions including hierarchical DocMRC modeling, multi‑teacher knowledge distillation, and instruction decomposition for efficient, high‑quality answers.
This article explains the AI model‑compression technique of knowledge distillation, describing how a large teacher network transfers its soft predictions to a lightweight student network using temperature‑scaled softmax, enabling deployment on resource‑constrained devices.
ConaCLIP introduces a fully‑connected knowledge interaction graph to distill large dual‑encoder models into compact ones, enhancing text‑image retrieval accuracy and efficiency on edge devices, with extensive experiments and supervision strategies demonstrating significant gains over existing baselines.