BestHub
Sign in
Tagged articles

Pretraining

128 articles · Page 2 of 2
Meituan Technology Team
Meituan Technology Team
Dec 2, 2021 · Artificial Intelligence

Pretraining Techniques for Search Advertising Relevance at Meituan

Meituan improves search‑ad relevance by applying pre‑trained BERT models enhanced with data‑augmented samples, multi‑task learning, keyword extraction and two‑stage knowledge distillation, producing a lightweight distilled model that, when fused with traditional relevance signals, boosts CTR, lowers Badcase@5 and raises NDCG while preserving revenue.

BERTPretrainingadvertising relevance
0 likes · 30 min read
Pretraining Techniques for Search Advertising Relevance at Meituan
JD Retail Technology
JD Retail Technology
Nov 16, 2021 · Artificial Intelligence

Automatic Product Copywriting for E-Commerce: The APCG System and Its AI Innovations

The APCG system, awarded the AAAI 2022 Innovation Application Prize, automatically generates e‑commerce product copy using a Transformer‑Pointer network and a pretrained sequence‑to‑sequence model, incorporates quality control, employs novel pretraining tasks, and has produced millions of descriptions that boost CTR, CVR, and GMV.

AINatural Language GenerationPretraining
0 likes · 6 min read
Automatic Product Copywriting for E-Commerce: The APCG System and Its AI Innovations
DataFunSummit
DataFunSummit
Nov 14, 2021 · Artificial Intelligence

Overview of Pre‑training Models and the UER‑py Framework for Natural Language Processing

This article introduces the importance of pre‑training in natural language processing, reviews classic pre‑training models such as Skip‑thoughts, BERT, GPT‑2 and T5, presents the modular UER‑py framework and its Chinese resources, compares it with Huggingface Transformers, and outlines practical deployment steps in industry.

NLPPretrainingUER-py
0 likes · 22 min read
Overview of Pre‑training Models and the UER‑py Framework for Natural Language Processing
DataFunTalk
DataFunTalk
Oct 22, 2021 · Artificial Intelligence

Applying AI Techniques to Credit Reporting and Risk Modeling: Model Structure, Pre‑training, Ranking and Interpretability

This article presents a comprehensive overview of how AI technologies are applied to credit reporting and loan risk modeling, detailing data characteristics, end‑to‑end model architectures, pre‑training strategies, risk‑ranking methods, and interpretability techniques for financial risk assessment.

AIPretrainingRisk Ranking
0 likes · 17 min read
Applying AI Techniques to Credit Reporting and Risk Modeling: Model Structure, Pre‑training, Ranking and Interpretability
Alimama Tech
Alimama Tech
May 27, 2021 · Artificial Intelligence

Explicit Semantic Cross Feature Learning via Pre-trained Graph Neural Networks for CTR Prediction (PCF‑GNN)

PCF‑GNN builds a heterogeneous graph of feature nodes and learns edge statistics via pre‑training, enabling it to infer unseen cross‑features, reduce storage by over 50%, and consistently improve CTR prediction AUC compared to implicit and explicit baselines, with proven online gains.

Graph Neural NetworkPretrainingcross feature
0 likes · 12 min read
Explicit Semantic Cross Feature Learning via Pre-trained Graph Neural Networks for CTR Prediction (PCF‑GNN)
DataFunTalk
DataFunTalk
May 8, 2021 · Artificial Intelligence

Attribute‑Level Sentiment Analysis for E‑commerce: Tasks, Challenges, and System Design

This article presents a comprehensive overview of sentiment analysis in user‑generated content, detailing document‑, sentence‑, and aspect‑level tasks, defining the Aspect Sentiment Triplet Extraction problem for e‑commerce reviews, describing a three‑stage pipeline with pre‑training, multi‑domain modeling and attribute normalization, and reporting significant business improvements such as 400% CTR lift, while also discussing data imbalance, annotation scarcity, and future research directions.

Natural Language ProcessingPretrainingSentiment Analysis
0 likes · 15 min read
Attribute‑Level Sentiment Analysis for E‑commerce: Tasks, Challenges, and System Design
Cyber Elephant Tech Team
Cyber Elephant Tech Team
Apr 28, 2021 · Artificial Intelligence

Understanding BERT: From Encoder-Decoder to Transformer and Attention

This article explains the BERT model by first reviewing the Encoder-Decoder framework, then detailing the attention mechanism—including self-attention and multi-head attention—before describing the full Transformer architecture and finally outlining BERT’s encoder-only design, training stages, and fine-tuning applications.

AttentionBERTEncoder-Decoder
0 likes · 15 min read
Understanding BERT: From Encoder-Decoder to Transformer and Attention
DataFunTalk
DataFunTalk
Apr 7, 2021 · Artificial Intelligence

Alibaba's Advances in Multilingual Neural Machine Translation: Research and Practice

This article presents Alibaba's comprehensive research on multilingual neural machine translation, covering motivations, model architectures, intermediate language modules, data‑augmentation strategies such as repair translation, integration of pre‑trained models with adapters, and engineering optimizations that enable a production‑ready system supporting over 200 languages.

AdapterAlibabaNeural Machine Translation
0 likes · 21 min read
Alibaba's Advances in Multilingual Neural Machine Translation: Research and Practice
DataFunTalk
DataFunTalk
Apr 5, 2021 · Artificial Intelligence

Summary of Methods and Findings from the NLP Chinese Pre‑training Model Generalization Challenge

The article reviews the Chinese NLP pre‑training model generalization competition, detailing data preprocessing, augmentation, external data usage, model scaling and architecture tweaks, loss functions, learning‑rate and adversarial training strategies, regularization techniques, post‑processing optimizations, and ineffective methods, highlighting their impact on performance metrics.

Data AugmentationLoss FunctionsNLP
0 likes · 15 min read
Summary of Methods and Findings from the NLP Chinese Pre‑training Model Generalization Challenge
DataFunTalk
DataFunTalk
Feb 20, 2021 · Artificial Intelligence

Industrial-Scale Machine Translation at Bytedance: Applications, Demos, and Research Advances

This article presents Bytedance's industrial machine‑translation platform, describing its global deployment, diverse product demos, underlying sequence‑to‑sequence models, BERT‑enhanced training strategies, prune‑tune sparsity techniques, multilingual pre‑training, document translation, and a high‑performance inference engine.

BERTPretrainingmachine translation
0 likes · 19 min read
Industrial-Scale Machine Translation at Bytedance: Applications, Demos, and Research Advances
Sohu Tech Products
Sohu Tech Products
Feb 17, 2021 · Artificial Intelligence

Improving BERT Pre‑training with RealFormer: Principles, Implementation, and Empirical Evaluation

This article analyzes the RealFormer modification to the Transformer architecture, details its implementation in BERT, and presents extensive experiments showing that while RealFormer can boost performance on low‑label‑count classification tasks, its benefits diminish or disappear as the number of classes grows.

AttentionBERTPretraining
0 likes · 12 min read
Improving BERT Pre‑training with RealFormer: Principles, Implementation, and Empirical Evaluation
DataFunTalk
DataFunTalk
Dec 25, 2020 · Artificial Intelligence

Exploring Pretraining Model Optimization and Deployment Challenges in NLP

This article reviews the evolution of pretraining models in NLP, discusses the practical challenges of deploying large models such as inference latency, knowledge integration, and task adaptation, and presents Xiaomi’s optimization techniques including knowledge distillation, low‑precision inference, operator fusion, and multi‑granularity segmentation for dialogue systems.

BERTDialogue SystemsNLP
0 likes · 15 min read
Exploring Pretraining Model Optimization and Deployment Challenges in NLP
Sohu Tech Products
Sohu Tech Products
Nov 4, 2020 · Artificial Intelligence

Understanding BERT: Architecture, Pre‑training, Fine‑tuning and Applications in Modern NLP

This article provides a comprehensive overview of BERT and related NLP advances, covering its historical context, model architecture, input‑output mechanisms, comparisons with CNNs, word‑embedding evolution, pre‑training strategies like MLM and next‑sentence prediction, and practical guidance for fine‑tuning and feature extraction.

BERTFine-tuningNLP
0 likes · 17 min read
Understanding BERT: Architecture, Pre‑training, Fine‑tuning and Applications in Modern NLP
JD Cloud Developers
JD Cloud Developers
Nov 4, 2020 · Artificial Intelligence

Multimodal AI Breakthroughs Unveiled at NLPCC 2020 Workshop

The article recaps the inaugural Multimodal Natural Language Processing workshop at NLPCC 2020, highlighting breakthroughs in multimodal summarization, pre‑training models, AI‑driven art, visual‑language interaction, and multimodal dialogue systems, and showcases research from leading institutions and industry partners.

AIMultimodalNLP
0 likes · 9 min read
Multimodal AI Breakthroughs Unveiled at NLPCC 2020 Workshop
DataFunTalk
DataFunTalk
Sep 23, 2020 · Artificial Intelligence

From Word Embedding to BERT: A Comprehensive Overview of Pre‑training Model Development in NLP

This article surveys the evolution of pre‑training models for natural language processing, detailing model architectures such as Encoder‑AE, Decoder‑AR, Encoder‑Decoder, Prefix LM, and PLM, analyzing why models like RoBERTa, T5, and GPT‑3 excel, and offering practical guidance for building strong pre‑training systems.

BERTNLPPretraining
0 likes · 47 min read
From Word Embedding to BERT: A Comprehensive Overview of Pre‑training Model Development in NLP
58 Tech
58 Tech
Aug 14, 2020 · Artificial Intelligence

Using SPTM in qa_match for the 58 City AI Competition: Data Preparation, Model Training, and Prediction

This article provides a step‑by‑step guide on preparing data, pre‑training the SPTM lightweight model, fine‑tuning a text‑classification model with qa_match, and generating competition‑ready predictions for the 58 City AI Algorithm Contest, including all required shell commands and parameter explanations.

AICompetitionPretraining
0 likes · 9 min read
Using SPTM in qa_match for the 58 City AI Competition: Data Preparation, Model Training, and Prediction
NetEase Media Technology Team
NetEase Media Technology Team
Jul 24, 2020 · Artificial Intelligence

Survey of Video Action Recognition Algorithms: 3D and 2D Convolutional Networks and Pre‑training

This survey reviews video action recognition, comparing 3D convolutional networks that jointly model spatial‑temporal cues but are computationally heavy with 2D‑based approaches like TSM and TIN that embed temporal shifts efficiently, and emphasizes how large‑scale pre‑training markedly improves performance despite limited labeled data.

2D convolutional networks3D convolutional networksPretraining
0 likes · 13 min read
Survey of Video Action Recognition Algorithms: 3D and 2D Convolutional Networks and Pre‑training
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 2, 2020 · Artificial Intelligence

How FashionBERT Boosts E‑Commerce Image‑Text Matching with Patch Embeddings

This article introduces FashionBERT, a multimodal BERT‑based model that replaces ROI‑based image tokens with uniform image patches to overcome e‑commerce specific challenges, details its architecture, adaptive loss balancing, deployment in Alibaba search, and reports significant performance gains on public and internal datasets.

BERTImage-Text MatchingMultimodal
0 likes · 13 min read
How FashionBERT Boosts E‑Commerce Image‑Text Matching with Patch Embeddings
DataFunTalk
DataFunTalk
Dec 27, 2019 · Artificial Intelligence

NLP Challenges and Tagging Solutions in Sina Weibo Feed

This article reviews the specific NLP difficulties encountered in Sina Weibo's feed—such as short text, informal language, and ambiguous user behavior—and details the multi‑stage tagging system, material library, multimodal modeling, multi‑task learning, and large‑scale pre‑training techniques used to address them.

BERTNLPPretraining
0 likes · 15 min read
NLP Challenges and Tagging Solutions in Sina Weibo Feed
Meituan Technology Team
Meituan Technology Team
Nov 14, 2019 · Artificial Intelligence

MT-BERT: Pre‑training and Fine‑tuning Practices at Meituan‑Dianping

MT‑BERT at Meituan‑Dianping combines mixed‑precision, domain‑adapted continual pre‑training, knowledge‑graph‑aware masking, and extensive compression techniques to produce fast, accurate BERT models that power fine‑grained sentiment analysis, intent classification, recommendation reasoning, and other NLP tasks across the platform.

BERTMT-BERTNLP
0 likes · 33 min read
MT-BERT: Pre‑training and Fine‑tuning Practices at Meituan‑Dianping
JD Tech Talk
JD Tech Talk
Nov 5, 2019 · Artificial Intelligence

GeoBERT: A Multi‑Task Pre‑trained Language Model for Chinese Address Text

This article introduces GeoBERT, a novel pre‑training method for Chinese address strings that leverages seven jointly constrained tasks to capture spatial semantics, administrative hierarchy, and similarity relationships, enabling downstream address classification, segmentation, POI extraction, similarity comparison, and authenticity verification with reduced annotation dependence.

Chinese languageGeoBERTGeocoding
0 likes · 15 min read
GeoBERT: A Multi‑Task Pre‑trained Language Model for Chinese Address Text
Alibaba Cloud Developer
Alibaba Cloud Developer
Aug 15, 2019 · Artificial Intelligence

How Auto Risk Transforms Behavior Sequence Data with Unsupervised Pre‑Training

This article introduces Auto Risk, a deep‑learning risk model for behavior‑sequence data that leverages unsupervised pre‑training with proxy tasks, details its convolution‑attention encoder, demonstrates significant gains across multiple business scenarios, and highlights its strong small‑sample and analogy capabilities.

PretrainingRisk ModelingUnsupervised Learning
0 likes · 20 min read
How Auto Risk Transforms Behavior Sequence Data with Unsupervised Pre‑Training
Alibaba Cloud Developer
Alibaba Cloud Developer
Jul 30, 2019 · Artificial Intelligence

Auto Risk: Pretraining Deep Models on Unlabeled Behavior Sequences

This article introduces Auto Risk, a behavior‑sequence deep‑learning framework that uses unsupervised pre‑training with proxy tasks to learn universal feature representations from massive unlabeled data, achieving significant gains in risk‑control scenarios, improving AUC, supporting multi‑scene generalization and small‑sample learning.

PretrainingRisk ModelingUnsupervised Learning
0 likes · 20 min read
Auto Risk: Pretraining Deep Models on Unlabeled Behavior Sequences
DataFunTalk
DataFunTalk
Jun 23, 2019 · Artificial Intelligence

Understanding XLNet: Differences from BERT, Innovations, and Experimental Analysis

This article examines XLNet, contrasting it with BERT by detailing its novel permutation language modeling, dual‑stream attention, and larger pre‑training data, and analyzes experimental results that show XLNet’s superior performance on reading‑comprehension, GLUE, and other NLP tasks, especially for long documents.

BERTNLPPermutation Language Model
0 likes · 27 min read
Understanding XLNet: Differences from BERT, Innovations, and Experimental Analysis
Hulu Beijing
Hulu Beijing
Apr 4, 2019 · Artificial Intelligence

How BERT, GPT, and ELMo Revolutionize Language Feature Representation

Natural language processing, a cornerstone of AI, relies on language models to capture linguistic features; this article reviews classic pre‑training models—ELMo, GPT, and BERT—explaining their architectures, training objectives, and how they boost downstream NLP tasks despite data‑scarcity challenges.

BERTELMoGPT
0 likes · 10 min read
How BERT, GPT, and ELMo Revolutionize Language Feature Representation
Meituan Technology Team
Meituan Technology Team
Jan 25, 2019 · Artificial Intelligence

Fine-grained User Review Sentiment Classification: AI Challenger 2018 Champion's Approach

Cheng Huige’s winning AI Challenger 2018 solution treated fine‑grained Chinese review sentiment as a 20‑aspect multi‑class task, combining a high‑capacity LSTM encoder with self‑attention, word‑and‑character embeddings, simplified ELMo pre‑training, diverse tokenizations and a weighted seven‑model ensemble (including BERT), which together delivered the competition’s top F1 performance.

BERTELMoNLP
0 likes · 14 min read
Fine-grained User Review Sentiment Classification: AI Challenger 2018 Champion's Approach