The article explains that the Transformer, introduced by Google in the 2017 "Attention Is All You Need" paper, replaces RNN/CNN with self‑attention, enabling full parallel computation, solving sequential bottlenecks and long‑range dependency issues that power today’s large AI models.
The article reviews MIT’s Supervised Memory Training (SMT) and its DAgger extension (DMT), which replace traditional back‑propagation through time with a Transformer‑based teacher, enabling one‑step memory supervision for RNNs, achieving parallel‑friendly training and superior long‑sequence performance on synthetic benchmarks, TinyStories and pixel‑wise image generation.
Google Research introduces Memory Caching (MC), a technique that gives RNNs growing memory capacity, bridging the gap with Transformers to enable ultra‑long context processing while reducing memory demands, and demonstrates its effectiveness through extensive language‑modeling and recall experiments.
This tutorial explains the nature of sequential data, why traditional feed‑forward networks struggle with it, and how recurrent architectures such as RNN, LSTM, and GRU capture temporal dependencies, complete with mathematical foundations, training algorithms, and full PyTorch implementations for sentiment analysis, text generation, and encoder‑decoder models.
This article provides a comprehensive overview of natural language processing, tracing its evolution from early symbolic and statistical stages through deep learning breakthroughs, detailing sequence models, key NLP tasks, text representation methods, and the development of modern architectures like RNN, LSTM, GRU, Transformer, and GPT series.
This article walks you through the evolution from basic mathematical functions to modern large‑scale models, explaining activation functions, forward and backward propagation, loss calculation, gradient descent, regularization, dropout, word embeddings, RNNs, and the core mechanics of the Transformer architecture.
This report details a university big‑data project that built a full pipeline—from raw petrochemical market data and text mining to variable selection, XGBoost/LightGBM/Lasoo regression and RNN/LSTM/GRU models—to predict product prices across multiple horizons, evaluate errors, and deliver an interactive demo.
This article outlines the four dominant generative AI architectures—Transformers, diffusion models, GANs, and RNNs—explaining their core mechanisms, key capabilities, and typical application domains such as chatbots, image creation, deep‑fake media, and time‑series analysis, helping readers choose the right model for their needs.
This article explains how recurrent neural networks (RNNs) add memory to neural models, details the gate mechanisms of LSTM and GRU, compares their structures and parameter counts, and illustrates their use in speech recognition, translation, stock prediction, and video generation, while highlighting practical insights and energy considerations.
This article explains why Transformer models surpass traditional RNN‑based seq2seq architectures by introducing self‑attention, multi‑head attention, and positional encoding, detailing the inner workings of encoders, decoders, and attention mechanisms, and comparing their advantages and limitations across NLP and vision tasks.
This article provides a friendly, step‑by‑step overview of five core deep‑learning architectures—CNN, RNN, GAN, Transformers, and encoder‑decoder—explaining their structures, key components, and typical use cases in image and natural‑language processing.
This article provides a thorough overview of large language models, explaining the relationship between NLP and LLMs, the evolution from RNN to LSTM, the fundamentals of attention mechanisms, and the architecture and operation of Transformer models, all illustrated with clear examples and diagrams.
This article presents a comprehensive overview of the RWKV large language model, covering its origin, attention‑free RNN architecture, performance benchmarks, evolution through v4 and v5, training pipelines, diverse application cases, open‑source ecosystem, and a detailed Q&A session.
This article introduces word‑level representations—from one‑hot encoding to dense word embeddings via Word2Vec—explains cosine similarity, then walks through the structure, limitations, and PyTorch implementation of a vanilla RNN, including a custom forward function and verification against the library API.
This article describes how a Chinese bank’s software development team leveraged AI techniques—text clustering and recurrent neural networks—to automatically classify and diagnose test-environment problems, dramatically reducing manual effort, improving issue visibility, and enabling self-healing mechanisms for faster, more reliable software delivery.
This article explains the challenges of processing sequential data, introduces LSTM as a solution to long‑term dependency problems in RNNs, details its cell state and gate mechanisms, showcases its architecture, and provides Python code examples for time‑series forecasting using Keras.
This article provides a comprehensive overview of deep neural network families, introducing twelve major architectures—including Feedforward, CNN, RNN, LSTM, DBN, GAN, Autoencoder, Residual, Capsule, Transformer, Attention, and Deep Reinforcement Learning—explaining their principles, structures, training methods, and offering Python/TensorFlow/PyTorch code examples.
This article introduces the basic architectures of recurrent neural networks (RNN), LSTM, and GRU, explains PyTorch APIs such as nn.RNN, nn.LSTM, nn.GRU, details their parameters, demonstrates code examples for building and testing these models, and provides practical insights for deep learning practitioners.
This article explains the fundamentals of Recurrent Neural Networks and Long Short‑Term Memory units, their gating mechanisms, and demonstrates a practical Python Keras example that predicts future PM2.5 concentrations using an LSTM model.
The paper proposes clustering thousands of related time series and training separate global LSTM models for each cluster, showing that this reduces heterogeneity, leverages shared information, and improves forecasting accuracy compared to individual models, with extensive experiments on CIF2016 and NN5 datasets.
This article introduces neural networks, covering their layered structure, common types like CNNs and RNNs, key components such as activation functions, loss, learning rate, backpropagation, dropout, batch normalization, and extends to reinforcement learning concepts including MDPs, policies, value functions, and Q‑learning.
This article explains the structure and operation of recurrent neural networks (RNNs), their limitations, how long short‑term memory (LSTM) networks overcome these issues with gated mechanisms, and provides a complete Python implementation for time‑series airline passenger forecasting.
This article introduces recurrent neural networks (RNNs) and their ability to handle sequential data, explains the limitations of vanilla RNNs, presents the LSTM architecture with its gates, and provides complete Keras code for data loading, model building, and training both vanilla RNN and LSTM models.
This presentation details how JD applied reinforcement learning—using RNN‑based user state modeling and a DDPG framework—to improve e‑commerce search ranking by optimizing long‑term cumulative value, showing significant offline and online gains in conversion and GMV.
This article surveys sequential recommendation methods, covering standard models such as pooling, RNN, CNN, attention, and Transformer, as well as long‑short term, multi‑interest, multi‑behavior approaches, and recent advances like contrastive learning, highlighting their impact on recommendation performance.
This article provides a comprehensive overview of sequence labeling tasks in NLP, covering their definition, common tag schemes (BIO, BIEO, BIESO), comparisons with other NLP tasks, major modeling approaches such as HMM, CRF, RNN and BERT, real‑world applications like POS tagging, NER, event extraction and gene analysis, and a step‑by‑step PyTorch implementation with dataset preparation, training pipeline, and evaluation metrics.
This article explains the fundamentals of seq2seq neural machine translation models, covering encoder‑decoder architecture, word embeddings, context vectors, RNN processing, and the attention mechanism introduced by Bahdanau and Luong, with visual illustrations and reference links for deeper study.
This article presents a comprehensive English summary of the deep learning specialization infographics originally shared by Andrew Ng, covering fundamentals, logistic regression, shallow and deep neural networks, regularization, optimization, hyperparameters, convolutional and recurrent networks, and practical advice for model building and evaluation.
The article describes Ctrip's payment fraud risk characteristics, a comprehensive automated iterative anti‑fraud model framework—including variable system, GAN‑augmented sample generation, RNN behavior encoding, and tree‑based classifiers—and demonstrates how this approach restores recall performance compared with traditional static models.
This article details how Amap leverages deep‑learning techniques—including temporal and auxiliary feature engineering, multi‑stage RNN models, Wide&Deep architectures, and an Attention‑TCN approach—to accurately identify and handle expired points of interest, improving map freshness and user experience.
The project develops a deep‑learning pipeline for Amap’s expired POI detection that integrates two‑year temporal trend features, industry and verification attributes, a variable‑length LSTM, a Wide‑Deep architecture, and a Wide‑Attention Temporal Convolutional Network, achieving higher accuracy and efficiency while outlining future macro‑and micro‑level enhancements.
This article explains knowledge tracing and its deep learning variant DKT, detailing the underlying RNN/LSTM models, data encoding, loss functions, experimental results, practical applications in adaptive learning, as well as advantages, limitations, and future research directions.
This article provides an in-depth overview of sequential recommendation, defining the problem, discussing data preparation, and reviewing various neural architectures—including MLP, CNN, RNN, Temporal CNN, self‑attention, and reinforcement‑learning approaches—while offering practical guidance on model selection and implementation.
This article provides a comprehensive overview of deep learning fundamentals, covering neural network basics, forward and backward feedback architectures, key models such as MLP, CNN, RNN, LSTM and GRU, training techniques like gradient descent, learning rate schedules, momentum, weight decay, and batch normalization.
During the 2023 Lunar New Year, Taobao Live launched the real‑time interactive game ‘Guess‑Draw Treasure’, which lets users sketch on mobile devices and have AI instantly recognize their drawings to win cash rewards; this article reveals the underlying AI techniques, challenges, model choices, datasets, and future plans.
This article walks through the development of deep learning for natural language processing, starting with basic neural cells and shallow networks, then exploring CNNs, RNNs, LSTMs, TextCNN, ESIM, ELMo, and culminating with the Transformer‑based BERT model, its training objectives, fine‑tuning strategies, and performance comparisons.
This article explores how deep learning techniques such as RNN, LSTM, 3D‑CNN, and CNN‑RNN hybrids can be applied to predict network bandwidth and traffic, comparing traditional time‑series methods with modern AI approaches and highlighting the potential of graph neural networks for future improvements.
This article explains how to use machine‑learning techniques—including data preprocessing, N‑gram, CBOW, and various RNN/LSTM models—to automatically recommend the next function in a test‑case step sequence, improving writing speed and efficiency for developers.
This article introduces Advanced LSTM (A‑LSTM), which linearly combines multiple past hidden states to overcome traditional LSTM's one‑step dependency, and demonstrates its application in utterance‑level speech emotion recognition, achieving a 5.5% accuracy improvement through attention‑based weighted‑pooling RNNs and auxiliary speaker and gender tasks.
The article dissects Google’s Quickdraw sketch‑recognition model, detailing its 1‑D convolutional front‑end, Bi‑LSTM encoder, and softmax classifier, explaining the TFRecord‑based normalization and interpolation steps, why pooling harms accuracy, and how the massive dataset can fuel diverse sequential‑learning applications and product concepts.
Traditional e‑commerce ranking ignores interactions among displayed items, but this study introduces a novel global ranking method that models mutual influences, optimizes expected GMV using extended global features and RNN‑based sequence generation, achieving a 5% GMV lift in large‑scale A/B tests.
The paper introduces JUMP, a novel three‑layer RNN architecture that simultaneously predicts click‑through rates and user dwell time in session‑based recommendation scenarios, leveraging a fast‑slow layer to handle short sessions, an attention layer to filter noise, and survival‑analysis‑based modeling of stay duration, achieving superior performance across multiple benchmark datasets.
Researchers introduce a novel suffix‑prediction mechanism for neural machine translation that separately generates stems and suffixes during decoding, dramatically reducing out‑of‑vocabulary errors and morphological mistakes in English‑Russian translation, achieving consistent improvements across RNN and Transformer models on large‑scale news and e‑commerce datasets.
This article provides a comprehensive guide to text classification, covering traditional pipelines, bag‑of‑words and TF‑IDF features, dimensionality‑reduction techniques, word‑embedding models such as GloVe, word2vec and fastText, and modern deep‑learning architectures like CNN, RCNN and HAN.
This article explains why machine learning is essential for complex tasks, defines neural networks, outlines three reasons to study them, and provides concise overviews of eight fundamental neural network architectures—including perceptron, CNN, RNN, LSTM, Hopfield, Boltzmann machines, deep belief networks, and deep autoencoders—grouped by their structural categories.
This article explains why attention mechanisms were introduced into Seq2Seq models, how they address the limitations of fixed‑length encoding, the role of bidirectional RNNs, and showcases their impact on machine translation and image captioning with illustrative diagrams.
This article explains how LSTM networks overcome RNN limitations by using input, forget, and output gates with sigmoid and tanh activations, describes the core update equations, discusses alternative activation functions and hard‑gate variants, and provides references for deeper study.
This article explains what Recurrent Neural Networks are, how they create text representations, why they suffer from gradient vanishing or explosion, and what architectural improvements such as LSTM, GRU, residual connections and gradient clipping can do to overcome these issues.
This article explains how Baidu Waimai's risk‑control team uses recurrent neural networks, especially LSTM, within TensorFlow to detect fraudulent merchants and users, compares static and dynamic RNN implementations, demonstrates a MNIST digit‑recognition example, and discusses optimization algorithms and model trade‑offs for real‑time fraud detection.
This article provides a comprehensive introduction to machine learning and deep learning, covering their definitions, classifications, key algorithms, neural network structures, core concepts such as generalization and regularization, and typical architectures like CNN and RNN, illustrated with numerous diagrams.
This article reviews how Liulishuo applied TensorFlow to implement a Deep Knowledge Tracing (DKT) student model for an adaptive learning system, covering the problem background, model architecture, TensorFlow implementation details, multi‑GPU training, and practical deployment considerations.
This article provides a comprehensive guide to Long Short-Term Memory networks, covering their motivation, detailed forward and backward equations, gate mechanisms, training algorithm, gradient checking, and a full C++ implementation, while also introducing the simpler GRU variant.
This article explains the fundamentals of recurrent neural networks (RNNs), their role in language modeling, various RNN architectures such as bidirectional and deep RNNs, the back‑propagation through time (BPTT) training algorithm, gradient challenges, vectorization techniques, and provides a step‑by‑step code implementation.
The article reviews symbolic knowledge bases such as WordNet, ConceptNet and FrameNet, explains how deep learning replaces them with vector‑based semantic representations, and discusses encoder‑decoder RNNs, attention mechanisms, and future directions for truly understanding language through experiential learning.
