This article presents a comprehensive overview of Alibaba Health's interactive BERT approach for improving relevance in health e‑commerce search, covering business background, model design, domain‑specific data construction, knowledge‑distilled twin‑tower deployment, experimental results, and a detailed Q&A session.
This article presents Xiaomi’s e‑commerce recommendation research, addressing four key challenges—sample selection bias, data sparsity, delayed feedback, and knowledge inconsistency—by introducing the Entire Space Delayed Feedback with Cross‑Task Knowledge Distillation (ESDC) model, which combines causal inference, cross‑task distillation, twin networks, and uncertainty weighting to improve CVR prediction and achieve a 15% GMV lift over the baseline.
This article describes the production challenges of using BERT for large‑scale text classification at Zuoyebang, explores lightweight alternatives such as knowledge distillation, pruning and quantization, and details a teacher‑student‑active‑learning pipeline that trains a TextCNN model to match BERT performance while dramatically reducing GPU consumption and improving throughput.
This talk explores the challenges and opportunities of leveraging sparsity in machine learning models, covering fine‑grained and coarse‑grained sparsity, NVIDIA Ampere’s 2:4 structured sparsity, knowledge‑distillation techniques for converting unstructured to structured sparsity, and model compression strategies for generative adversarial networks.
This article describes Meituan Search's pre‑ranking (coarse‑ranking) system evolution and presents two major optimization strategies—leveraging knowledge distillation to align coarse‑ranking with fine‑ranking and employing neural architecture search to jointly improve effectiveness and latency—demonstrating significant offline and online performance gains.
Meituan’s search team upgraded its pre‑ranking layer from simple linear models to end‑to‑end neural networks, boosting effectiveness by applying three knowledge‑distillation techniques—including result‑list, score, and contrastive representation transfer—and by using latency‑aware neural architecture search to automatically select features and network structures, achieving significant recall and CTR gains without added latency.
The Cascade Architecture for Creative Selection (CACS) reorders the advertising pipeline by placing a dual‑tower creative‑selection module ahead of ranking, using soft‑label list‑wise distillation and adaptive dropout to jointly optimize creatives and ads, yielding 5% latency increase but significant CTR and RPM gains in Taobao’s search ads.
Knowledge Distillation, a teacher‑student model compression technique, enables large, high‑performing deep neural networks to transfer their learned representations to smaller models, achieving comparable accuracy with faster inference, reduced resource consumption, and broader applicability in computer‑vision tasks.
The paper introduces UKD, an uncertainty‑regularized knowledge‑distillation framework that uses a click‑adaptive teacher to generate pseudo‑conversion labels for unclicked impressions and trains a student model with uncertainty‑weighted loss, thereby mitigating sample‑selection bias and achieving up to 3.4% CVR improvement and 4.3% CPA reduction on large‑scale advertising datasets.
The 2021 Taobao New‑Product team boosted e‑commerce recommendation by redesigning the coarse‑ranking stage with a dual‑tower DSSM, low‑cost feature‑crossing, NOVA attention and multi‑task distillation from a fine‑ranking teacher, delivering up to +30‰ GAUC gain and 3‑5 % online CTR and click improvements.
This article presents a comprehensive overview of inference optimization methods—including model pruning, quantization, knowledge distillation, caching, instruction‑set acceleration, and operator fusion—and details a GPU‑centric parallel acceleration methodology with CUDA basics, performance‑analysis tools, theoretical limits, and practical case studies, all illustrated with real‑world examples from Tencent's intelligent dialogue products.
We applied knowledge distillation to a video query‑document BERT matcher, compressing the 12‑layer teacher into production‑ready 1‑layer ALBERT and tiny TextCNN students using combined soft, hard, and relevance losses plus AutoML‑tuned hyper‑parameters, achieving sub‑5 ms latency and up to 2.4% AUC improvement over the original model.
This article presents DRL-Rec, a distilled reinforcement learning framework for recommendation that integrates an exploring‑filtering module and confidence‑guided distillation to compress RL‑based recommenders while improving accuracy, and reports significant offline and online performance gains on a large‑scale system.
This article reviews three consecutive works from Alibaba DAMO Academy on compressing and distilling large pretrained language models—AdaBERT, L2A, and Meta‑KD—detailing their motivations, neural‑architecture‑search‑based designs, loss formulations, experimental results, and insights from a Q&A session.
This talk presents Alibaba DAMO Academy’s recent work on compressing large pretrained language models, covering task‑adaptive AdaBERT, data‑augmented L2A, and meta‑knowledge distillation Meta‑KD, describing their motivations, architectures, NAS‑based search, loss designs, and experimental results across multiple NLP tasks.
Meituan improves search‑ad relevance by applying pre‑trained BERT models enhanced with data‑augmented samples, multi‑task learning, keyword extraction and two‑stage knowledge distillation, producing a lightweight distilled model that, when fused with traditional relevance signals, boosts CTR, lowers Badcase@5 and raises NDCG while preserving revenue.
Alibaba's advertising platform combines knowledge distillation to score traffic, exposure forecasting via GBDT, and PID-based pacing to guarantee contracted impression volumes while improving CTR/CVR, handling delayed exposure and traffic selection, achieving near‑perfect delivery in large promotions.
The article provides a comprehensive technical overview of PaddleOCR's PP‑OCRv2, detailing its five major algorithmic enhancements, performance improvements over previous versions, historical milestones, core capabilities, and links to the open‑source repositories for developers interested in state‑of‑the‑art OCR solutions.
This article reviews BERT’s architecture, analyzes the storage and compute costs of each layer, and systematically presents compression methods—including quantization, pruning, knowledge distillation (Distilled BiLSTM and MobileBERT), and structure‑preserving techniques—aimed at enabling efficient deployment on resource‑constrained mobile devices.
This article examines the internal structure of BERT and systematically presents various model‑compression strategies—including quantization, pruning, knowledge distillation, and structure‑preserving techniques—highlighting their impact on storage, computational cost, and inference speed for deployment on resource‑constrained mobile devices.
The article describes the Hubble Intelligent Audience Platform’s three‑generation algorithmic evolution—starting from a DSSM‑based model, moving to an asynchronous GNN plus lightweight learning architecture, and finally integrating incremental learning with meta‑weighting—to improve audience expansion for mobile marketing campaigns.
The article surveys Amap’s AI‑driven mobility solutions—from personalized, multi‑objective route planning using Cell‑Based Routing and bias‑aware sorting, through spatio‑temporal ETA prediction and lightweight BERT‑based traffic‑event mining, to rapid POI freshness updates and a future global scheduling system that coordinates vehicles and signals via multi‑agent reinforcement learning.
This article describes the challenges of deploying a large BERT model as an online service for 360 Search and details engineering optimizations—including framework selection, model quantization, knowledge distillation, stream scheduling, caching, and dynamic sequence handling—that dramatically improve latency, throughput, and resource utilization.
This article details the engineering challenges of serving a large BERT model in real‑time for 360 Search and describes a series of optimizations—including TensorRT‑based kernel fusion, model quantization, knowledge distillation, multi‑stream execution, caching, and dynamic sequence handling—that together achieve low latency, high throughput, and stable deployment on GPU clusters.
This article details the practical upgrades of iQIYI's short‑video recommendation coarse‑ranking pipeline, moving from a GBDT model to a dual‑tower DNN, applying knowledge distillation, embedding compression, inference optimizations, and finally a cascade architecture to align with the fine‑ranking model while reducing resource consumption.
iQIYI’s short‑video recommendation team replaced a GBDT coarse‑ranking model with a lightweight dual‑tower DNN, applied knowledge distillation, sparse‑aware embedding optimization, and inference merging, then introduced a cascade MMOE architecture, achieving comparable accuracy with half the memory, ~19 ms latency reduction, and measurable gains in watch time, CTR and engagement.
This article presents a systematic exploration of text pre‑training models for dialogue classification in information‑security scenarios, comparing baseline TextCNN, an enhanced TextCNN_role, RoBERTa with domain‑adaptive pre‑training, and a distilled mini‑model, and discusses their performance, trade‑offs, and future directions.
This talk by Zhihu search algorithm engineer Shen Zhan details the evolution of text relevance models from TF‑IDF/BM25 to deep semantic matching and BERT, explains the challenges of deploying BERT at scale, and describes practical knowledge‑distillation techniques that improve both online latency and offline storage while maintaining search quality.
This article presents a comprehensive overview of Didi's machine translation platform, covering its evolution from statistical to neural models, the Transformer architecture with relative position and larger FFN, data preparation, training strategies such as back‑translation and knowledge distillation, deployment optimizations with TensorRT, and the team's successful participation in the WMT2020 news translation task.
This article reviews four mainstream model compression and acceleration methods—structural optimization, pruning, quantization, and knowledge distillation—explaining their principles, implementations, and performance, and presents practical examples such as DistillBERT, TinyBERT, and FastBERT with comparative results.
This article reviews the evolution of pretraining models in NLP, discusses the practical challenges of deploying large models such as inference latency, knowledge integration, and task adaptation, and presents Xiaomi’s optimization techniques including knowledge distillation, low‑precision inference, operator fusion, and multi‑granularity segmentation for dialogue systems.
Didi's machine translation system combines a Transformer‑big architecture with relative position representations, enlarged feed‑forward networks, iterative back‑translation, knowledge‑distillation and domain fine‑tuning, optimized via TensorRT for speed, achieving a BLEU 36.6 and third place in the WMT2020 Chinese‑to‑English news task.
The article reviews deep‑model compression methods—ADMM‑based structured pruning, low‑bit quantization, and teacher‑student knowledge distillation—and their automated AutoCompress workflow, demonstrating how these techniques shrink neural networks enough to run real‑time driver‑monitoring and other intelligent cockpit functions on resource‑limited automotive hardware while preserving accuracy.
This article examines how large pretrained BERT models can be compressed via knowledge distillation into a lightweight TextCNN classifier for efficient garbage opinion detection, detailing traditional distillation methods, several practical schemes, experimental results, and the advantages of the approach.
The Meituan Search team boosted ranking relevance by training a domain‑specific BERT, applying data augmentation, brand‑sample optimization, knowledge‑graph fusion, multi‑task and pairwise fine‑tuning, joint end‑to‑end training with LambdaLoss ranking models, and compressing the model for low‑latency inference, delivering up to +925 BP offline accuracy gains and measurable CTR and NDCG improvements in production.
This article reviews the motivations, challenges, and a comprehensive set of algorithmic, framework, and hardware methods—including structural optimization, quantization, pruning, and knowledge distillation—to compress and accelerate deep learning models for deployment on mobile devices, highlighting benefits such as reduced server load, lower latency, improved reliability, and enhanced privacy.
This article reviews how knowledge distillation—using a large teacher model to guide a smaller student model—can be applied across the recall, coarse‑ranking, and fine‑ranking stages of recommendation systems, detailing logits‑based and feature‑based approaches, joint and two‑stage training, and point‑wise, pair‑wise, and list‑wise loss designs.
Deep learning models often suffer from slow training and deployment due to their size, but a range of advanced acceleration methods—including model architecture optimization, pruning, quantization, knowledge distillation, and distributed training techniques—can dramatically improve speed and efficiency while maintaining performance.
This article describes iQIYI’s dual‑DNN ranking architecture that combines a high‑capacity teacher network with a lightweight student network via online knowledge distillation, addressing the trade‑off between model effectiveness and inference efficiency in large‑scale recommendation systems.
The article introduces iQIYI's double‑DNN ranking architecture that combines a high‑performance teacher network with a lightweight student network through online knowledge distillation, detailing the evolution of deep learning‑based ranking models, the motivation for model upgrades, training pipelines, and experimental results that demonstrate significant latency reduction and ROI improvement.
iQIYI’s dual‑DNN ranking model uses an online teacher‑student knowledge‑distillation framework where a complex teacher DNN shares representations with a lightweight student DNN, enabling end‑to‑end training, large‑scale feature crossing, and substantially higher recommendation accuracy while cutting inference latency and model size.
The paper presents UVA‑Net, a lightweight video‑attention network trained via coupled knowledge distillation, which matches the accuracy of eleven state‑of‑the‑art models while using only 0.68 MB of storage and achieving up to 10,106 FPS on GPU (404 FPS on CPU), thanks to a MobileNetV2‑based CA‑Res block and a teacher‑student framework that leverages low‑resolution inputs to drastically cut parameters and computational cost.
This article reviews why deep neural networks are over‑parameterized, outlines the challenges of deploying them on mobile and embedded devices, and presents six major strategies—pruning, low‑rank approximation, filter selection, quantization, knowledge distillation, and novel architecture design—to accelerate and compress models while preserving performance.
Rocket Launching introduces a novel co‑training framework that jointly trains a lightweight network and a more powerful booster network, sharing parameters and using gradient‑blocking and hint loss to improve click‑through‑rate prediction accuracy while keeping online inference latency unchanged, validated on public datasets and Alibaba’s ad system.
Online click‑through‑rate (CTR) prediction demands millisecond‑level response times, yet deep models are too slow; this paper introduces a “Rocket Launching” framework that jointly trains a lightweight net and a powerful booster net, sharing parameters and using gradient‑blocking and hint loss to improve accuracy without increasing inference latency.
