This article reviews the background and evolution of pre‑training models in NLP, introduces classic models such as Skip‑thoughts, BERT, and T5, and details the modular UER‑py framework, its comparison with HuggingFace Transformers, available Chinese pre‑trained weights, and practical deployment workflows.
The paper presents DeepDebug, a transformer‑based system that leverages large pretrained models and extensive synthetic and real‑world data to automatically localize and fix bugs in Python code, achieving significant improvements in patch generation success rates and reduction of false positives on benchmarks such as QuixBugs.
This article reviews modern text‑recognition systems, detailing how encoders such as CNN, CNN‑BiLSTM, and Transformer‑based models extract visual features, and how decoders like Position Attention, Transformer decoders, and RNN Seq2Seq align variable‑length text, while also discussing CTC loss and practical design choices.
VisTR redefines video instance segmentation as an end‑to‑end sequence‑to‑sequence task, using a CNN backbone, Transformer encoder‑decoder with instance queries, and Hungarian matching to jointly predict masks, classes, and tracks across frames, achieving state‑of‑the‑art accuracy (40.1 AP) and 57.7 FPS on YouTube‑VIS.
The article reviews the evolution of iQIYI's short‑video recommendation recall pipeline, detailing multi‑interest recall methods such as clustering‑based recall, MOE‑based recall, single‑activation multi‑interest networks, regularization strategies, dynamic capacity handling, and multimodal extensions, and discusses their impact on recommendation performance.
This article explains the BERT model by first reviewing the Encoder-Decoder framework, then detailing the attention mechanism—including self-attention and multi-head attention—before describing the full Transformer architecture and finally outlining BERT’s encoder-only design, training stages, and fine-tuning applications.
This article introduces a personalized re‑ranking model for recommendation systems, explaining the limitations of traditional point‑wise ranking, describing the PRM architecture with input, encoding, and output layers using multi‑head attention and pre‑trained personalization features, and presenting experimental results and future extensions.
This article presents a comprehensive overview of Alibaba's 1688 live‑streaming recommendation system, detailing core challenges such as heterogeneous behavior modeling, multi‑objective optimization, and bias mitigation, and describing four successive model iterations—from feature‑engineered GBDT to attention‑based heterogeneous networks and transformer architectures—along with experimental results and practical insights.
The talk reviews three major 2020 advances in computer vision—self‑supervised learning surpassing supervised pre‑training, the successful adoption of Transformer‑based attention models for detection and classification, and the emergence of Neural Radiance Fields for view synthesis—while highlighting related research from Microsoft Research Asia and the broader community.
User behavior sequence modeling, crucial for search and recommendation advertising ranking, has evolved from simple pooling to attention, RNN, capsule, and Transformer architectures, with industrial applications across e‑commerce, social, video, and music platforms, and future directions include time‑aware, multi‑dimensional, and self‑supervised approaches.
This article analyzes the RealFormer modification to the Transformer architecture, details its implementation in BERT, and presents extensive experiments showing that while RealFormer can boost performance on low‑label‑count classification tasks, its benefits diminish or disappear as the number of classes grows.
Learn how to create a low‑cost AI bot for the mobile game Honor of Kings by capturing the phone screen with scrcpy, generating action commands from game images using a Transformer model, and executing those commands via minitouch, complete with setup steps, required tools, and code links.
The 2017 Transformer model introduced by Vaswani et al. revolutionized machine translation by relying solely on attention mechanisms, outperforming traditional RNN and CNN approaches through parallel processing and improved contextual understanding.
This tutorial shows how to create a civilian‑grade AI for the mobile game Honor of Kings by mirroring the phone with scrcpy, generating action commands from game screenshots using a Transformer model, and executing them via minitouch on Android devices.
The qa_match open‑source tool from 58 Tongcheng, now at version 1.3, introduces semi‑automatic knowledge‑base mining for cold‑start and online scenarios and upgrades its Simple Pre‑trained Model (SPTM) with Transformer‑based feature representation to improve question‑answer matching performance.
This article introduces LightSeq, an open‑source, GPU‑accelerated inference engine that dramatically speeds up Transformer‑based models such as BERT and GPT by up to 14× over TensorFlow, supports multiple decoding strategies, integrates seamlessly with major deep‑learning frameworks, and provides detailed performance benchmarks and technical optimizations.
This article provides a comprehensive, illustrated walkthrough of OpenAI's GPT‑2 language model, covering its decoder‑only Transformer architecture, self‑attention mechanisms, token processing, training data, differences from BERT, and applications beyond language modeling, enriched with visual diagrams and code snippets for deeper understanding.
This article provides a step‑by‑step illustrated guide to the Transformer architecture, covering its macro structure, detailed self‑attention mechanisms, multi‑head attention, positional encoding, residual connections, decoder operation, training process, loss functions, and includes complete PyTorch and custom Python code examples.
This article provides a comprehensive overview of BERT and related NLP advances, covering its historical context, model architecture, input‑output mechanisms, comparisons with CNNs, word‑embedding evolution, pre‑training strategies like MLM and next‑sentence prediction, and practical guidance for fine‑tuning and feature extraction.
Didi's machine translation system combines a Transformer‑big architecture with relative position representations, enlarged feed‑forward networks, iterative back‑translation, knowledge‑distillation and domain fine‑tuning, optimized via TensorRT for speed, achieving a BLEU 36.6 and third place in the WMT2020 Chinese‑to‑English news task.
The second‑place team tackled KDD Cup 2020’s Multimodal Recall challenge by fine‑tuning ImageBERT and LXMERT on query‑image pairs, generating negatives, applying AMSoftmax and multi‑similarity losses, ensembling weighted predictions, and using score‑based post‑processing, boosting NDCG@5 to 0.8352 and powering Meituan’s multimodal search pipeline.
This article surveys the evolution of pre‑training models for natural language processing, detailing model architectures such as Encoder‑AE, Decoder‑AR, Encoder‑Decoder, Prefix LM, and PLM, analyzing why models like RoBERTa, T5, and GPT‑3 excel, and offering practical guidance for building strong pre‑training systems.
DiDi AI Labs’ NLP team earned third place in the WMT2020 Chinese‑to‑English news translation task with a 36.6 BLEU score, using an enhanced Transformer‑2 model that incorporates self‑attention, relative positional attention, iterative back‑translation, knowledge distillation, data cleaning, ensembling, and other techniques, now deployed across DiDi’s international services.
This article shares a competition winner's step‑by‑step solution for predicting user age and gender from ad click sequences, treating IDs as words, using word2vec embeddings, a custom transformer‑LSTM model, dual‑task loss, and weight‑search post‑processing.
This article provides a comprehensive technical overview of modern speech recognition, covering Didi’s driver‑assistant and smart‑customer‑service applications, fundamental ASR concepts, classic GMM‑HMM methods, deep‑learning breakthroughs such as DNN‑HMM, CTC, attention‑based and transformer models, practical training tricks, signal‑processing steps, and multimodal fusion techniques.
Meituan’s search ranking system integrates Transformer‑based models across feature engineering, behavior sequence modeling, and re‑ranking, adapting AutoInt‑style embeddings and multi‑stage attention mechanisms to boost QV_CTR and NDCG, while outlining future enhancements with BERT, graph neural networks, and reinforcement learning.
This article describes iQIYI's short‑video content tagging system, outlining the challenges of extracting type and abstract tags from multimodal data, detailing the evolution from text‑only models to image‑fusion, BERT‑enhanced, and video‑frame models, and discussing their applications and future directions.
This article proposes a method to identify low-quality text in business data by training a large-scale unsupervised language model to compute sentence perplexity, converting the detection problem into a threshold decision, and details model design, challenges, optimizations, and online performance results.
This article examines the challenges of short‑video content tagging and describes iQIYI's multi‑stage evolution from simple text‑only models to sophisticated multimodal architectures that fuse cover images, BERT embeddings, and video frames to improve tag generation accuracy.
iQIYI’s short‑video tagging system tackles multimodal fusion, open‑set and abstract tags by evolving from a text‑only model through cover‑image, BERT‑vector, and video‑frame fusion architectures, enabling automated labeling, personalized recommendation, and semantic search while planning to add OCR, audio, and knowledge‑graph enhancements.
This article systematically reviews the history, core concepts, variants, and practical implementations of attention mechanisms—from early additive and multiplicative forms to self‑attention, multi‑head attention, and recent transformer‑based models—highlighting why attention has become fundamental in modern AI research.
This article presents a Transformer‑based framework that simultaneously generates visually compatible outfits and personalizes recommendations by leveraging multimodal item embeddings and user behavior, achieving significant gains in compatibility prediction, fill‑in‑the‑blank accuracy, and click‑through rate on Alibaba's iFashion platform.
This article introduces KOBE, a knowledge‑based personalized text generation system that leverages Transformer architecture, attribute fusion, and external knowledge graphs to produce fluent, domain‑aware recommendation reasons for e‑commerce products, with a case study on the Spring Festival cloud theme.
This article systematically reviews the evolution of attention mechanisms—from early additive and multiplicative forms to self‑attention and multi‑head variants—explaining their core three‑step framework, key differences, and why they have become essential across NLP, vision, and broader AI applications.
This article reviews the historical development of natural language processing language models, covering expert rule‑based systems, statistical n‑grams, smoothing techniques, neural network models such as NNLM, RNN, word2vec, GloVe, ELMo, and the transformer‑based breakthroughs of GPT, BERT and GPT‑2, and summarizes their impact on modern NLP tasks.
This article surveys the evolution of machine translation from early rule‑based systems to modern neural architectures, explains how translation engines are trained, highlights recent advances such as attention and Transformers, and shares practical experience and current challenges in the field.
This article reviews the challenges of media‑domain named entity recognition, outlines the evolution from rule‑based methods through traditional machine‑learning and deep‑learning models to attention‑based Transformers, and details Sohu’s practical Bi‑LSTM‑CRF system with data‑annotation strategies and performance results.
This article reviews the history of natural language processing, explains key deep‑learning models such as NNLM, Word2vec, CNN, RNN, attention mechanisms, and Transformers, and discusses their applications, future trends, and practical considerations in NLP tasks.
This article reviews the current state of machine translation, explores the challenges of ambiguity, quality, and domain specificity, and presents human‑in‑the‑loop translation techniques—including attention‑enhanced models, transformer architectures, and online learning—while discussing practical productization and deployment considerations.
This article explains BERT’s architecture—from its bidirectional Transformer encoder and attention mechanisms to its pre‑training tasks—and presents extensive experiments showing its superior performance on various Chinese and English text‑classification benchmarks across multiple datasets.
This article provides a comprehensive, step‑by‑step explanation of the Transformer architecture, covering its encoder‑decoder structure, self‑attention, multi‑head attention, positional encoding, residual connections, and training processes, illustrated with diagrams and code snippets to aid readers new to neural machine translation.
This article explains how reinforcement learning, specifically the DDPG algorithm combined with Transformer-based networks, is applied to improve large‑scale news recall systems, detailing the business scenario, algorithm selection, model architecture, speed optimizations, training challenges, and observed online performance gains.
The article summarizes a video on Laravel project structuring, explaining how separating responsibilities into layers such as Repository for data access, Service for business logic, Presenter for view preparation, Transformer for data shaping, and Formatter for consistent API responses improves maintainability and scalability.
Researchers introduce a novel suffix‑prediction mechanism for neural machine translation that separately generates stems and suffixes during decoding, dramatically reducing out‑of‑vocabulary errors and morphological mistakes in English‑Russian translation, achieving consistent improvements across RNN and Transformer models on large‑scale news and e‑commerce datasets.
