Tagged articles
Self-Attention
67 articles · Page 1 of 1
This article introduces three complementary attention heatmap visualizations — image-to-text, text-to-image, and image-to-image — for interpreting text-to-image diffusion models, detailing aggregated attention computation, bilinear interpolation rendering, and an interactive Flask web service to diagnose prompt adherence and model biases.
Flask web serviceSelf-Attentionattention visualization
0 likes · 12 min read

The article explains why vanilla Transformers cannot sense word order, introduces sinusoidal positional encoding as the elegant solution, compares it with learned and relative encodings, and shows how modern variants like RoPE preserve both absolute and relative positions for long texts.
Learned EncodingPositional EncodingRelative Position
0 likes · 9 min read

The article explains how the Transformer architecture replaces RNNs with self‑attention, detailing the Q‑K‑V mechanism, positional encodings such as RoPE, multi‑head attention, modern improvements like SwiGLU and RMSNorm, and provides formulas for parameter and FLOP estimation.
FFNPositional EncodingRMSNorm
0 likes · 26 min read

The article explains that the Transformer, introduced by Google in the 2017 "Attention Is All You Need" paper, replaces RNN/CNN with self‑attention, enabling full parallel computation, solving sequential bottlenecks and long‑range dependency issues that power today’s large AI models.
AIAttention Is All You NeedGPT
0 likes · 3 min read

This article explains how modern AI large models—such as GPT‑4, Claude 3.5, Gemini, Llama, and DeepSeek—rely on the Transformer architecture, detailing token embeddings, self‑attention, positional encoding, and the three‑stage training pipeline of pre‑training, instruction fine‑tuning, and alignment optimization.
Model TrainingPositional EncodingSelf-Attention
0 likes · 5 min read

The article explains why modern interviewers ask about Transformer fundamentals, breaks down its core components such as self‑attention, multi‑head attention, feed‑forward networks, residual connections and positional encodings, and demonstrates a complete PyTorch toy model that predicts the sum‑mod‑10 of integer sequences while visualizing loss curves, attention heatmaps, embedding PCA and early‑stage gradient norms.
Gradient AnalysisModel VisualizationPyTorch
0 likes · 20 min read


CodePathJun 3, 2026 · Artificial Intelligence
The article dissects how the 2017 "Attention Is All You Need" paper sparked a fundamental redesign of sequence modeling by replacing recurrent and convolutional approaches with self‑attention, detailing its mathematical foundations, architectural components, training tricks, limitations, and emerging alternatives such as Mamba.
Attention MechanismMambaPositional Encoding
0 likes · 24 min read

The article explains that large language models are massive probability engines built on the Transformer architecture with self‑attention, trained through costly pre‑training on trillions of tokens, then refined by instruction fine‑tuning and RLHF, ultimately predicting the next token to generate text.
PretrainingRLHFSelf-Attention
0 likes · 5 min read

A developer recreated a 1970s PDP‑11 environment, wrote a single‑layer, single‑head Transformer in assembly, and trained it on a sequence‑reversal task, achieving 100% accuracy after about 350 steps and a total training time of roughly 5.5 minutes.
Low-resource AIPDP-11Self-Attention
0 likes · 16 min read

The article analyzes the hidden conflict between [CLS] and patch tokens in Vision Transformers, reveals how shared normalization and linear layers cause computational friction, and demonstrates that layer‑specific parameters dramatically improve dense prediction tasks without increasing inference FLOPs.
Computer VisionDense PredictionLayer Specialization
0 likes · 9 min read

This guide walks you through building, training, and fine‑tuning a Transformer‑based large language model entirely from scratch using PyTorch, covering tokenization, self‑attention, multi‑head attention, positional encoding, model architecture, data preparation, training loops, and fine‑tuning on custom lyrics.
GPTLLMPyTorch
0 likes · 43 min read


Qborfy AIFeb 21, 2026 · Artificial Intelligence
This article explains the self‑attention mechanism behind transformers, detailing its core components, mathematical formulation, step‑by‑step example, multi‑head extension, industry use cases, and a thorough comparison with RNN and CNN approaches, all supported by concrete numbers and citations.
Attention MechanismSelf-AttentionTransformer
0 likes · 8 min read

Vision‑language models turn continuous images into discrete tokens through patch extraction, encoding, and projection, enabling Transformers to reason jointly over vision and text, but this compression introduces limits in spatial reasoning, counting, and resolution sensitivity that users must understand.
Self-AttentionVision-Language Modelscounting
0 likes · 22 min read

This article provides a comprehensive, step‑by‑step walkthrough of the Transformer architecture, covering input embedding, positional encoding, the mechanics of Q‑K‑V attention, scaled dot‑product formulas, multi‑head and masked attention, feed‑forward networks, residual connections, layer normalization, decoder generation, and recent attention‑optimization techniques.
AttentionFeed-Forward NetworkPositional Encoding
0 likes · 39 min read

The paper introduces MLF, a multi‑cycle learning framework that integrates three novel modules—inter‑cycle redundancy filtering (IRF), learnable weighted integration (LWI), and multi‑cycle adaptive patch (MAP)—plus a patch‑squeeze component, achieving higher accuracy and efficiency on financial time‑series tasks such as fund‑sales prediction and outperforming strong single‑ and multi‑cycle baselines, with successful deployment in Alipay’s fund inventory system.
Alipay deploymentSelf-Attentionfinancial AI
0 likes · 16 min read

This article walks through the complete workflow of AI large language models, from turning user queries into token matrices via tokenization and embedding, through the Transformer’s self‑attention and multi‑head mechanisms, to decoding logits into human‑readable text, while also covering position encoding, long‑context strategies, generation parameters, and practical engineering tips.
Inference OptimizationSelf-AttentionTransformer
0 likes · 29 min read

This article explains the fundamental purpose, architecture, and variants of the Transformer model—including Encoder‑Decoder, Encoder‑only, and Decoder‑only designs—while detailing how attention mechanisms work and why modern large‑language models favor the Decoder‑only approach, providing a concise framework for answering interview questions.
AI InterviewEncoder-DecoderSelf-Attention
0 likes · 10 min read

This article explains the Transformer architecture from "Attention Is All You Need", covering self-attention, multi-head attention, layer normalization, positional encoding, and the encoder-decoder structure with PyTorch code, showing how Transformers enable parallel sequence processing without recurrence.
Attention Is All You NeedEncoder-DecoderLayer Normalization
0 likes · 10 min read

This tutorial walks through the fundamentals of ChatGPT by explaining language modeling, character‑level tokenization, data preprocessing pipelines, the evolution from simple bigram models to scaled dot‑product self‑attention, multi‑head mechanisms, full Transformer blocks, and how to train and generate Shakespeare‑style text with a GPT model.
ChatGPTGPTPython
0 likes · 50 min read

This guide breaks down the complex terminology of large language models—explaining tokens, transformers, self‑attention, RAG, scaling laws, dense vs. sparse architectures, and training stages—using clear analogies and step‑by‑step explanations so readers can confidently understand and work with modern AI systems.
AI FundamentalsModel TrainingRAG
0 likes · 35 min read

This guide breaks down the core logic, computation steps, formulas, and common interview questions about Self‑Attention and Multi‑Head Attention in Transformers, offering concrete explanations, dimensional examples, and practical answering techniques to help candidates ace large‑model algorithm interviews.
Interview TipsSelf-AttentionTransformer
0 likes · 8 min read


Qborfy AIAug 12, 2025 · Artificial Intelligence
This article explains how massive Transformer‑based large language models compress text data into mathematical representations, why scale, self‑attention, and training paradigms enable emergent general intelligence, and walks through tokenization, embedding, multi‑layer attention, architecture choices, energy costs, and hallucination mitigation.
AIEmbeddingLLM
0 likes · 6 min read

This article reviews the MPCT framework—a multiscale point‑cloud transformer built on a residual network that leverages permutation‑invariant self‑attention, point‑enhancement, and hierarchical feature aggregation to achieve state‑of‑the‑art results on ModelNet40 and ScanObjectNN datasets.
3D classificationSelf-Attentionmultiscale
0 likes · 14 min read


Qborfy AIAug 8, 2025 · Artificial Intelligence
This article explains how the Transformer model replaces sequential RNN processing with parallel self‑attention, detailing its core components, positional encoding, encoder‑decoder workflow, industry impact, and surprising facts such as training speed gains and energy efficiency.
AISelf-AttentionTransformer
0 likes · 5 min read

This article explains why Transformer models surpass traditional RNN‑based seq2seq architectures by introducing self‑attention, multi‑head attention, and positional encoding, detailing the inner workings of encoders, decoders, and attention mechanisms, and comparing their advantages and limitations across NLP and vision tasks.
AttentionGRULSTM
0 likes · 30 min read

This article demystifies the Transformer architecture for beginners, explaining its relationship to large models, the self‑attention and multi‑head attention mechanisms, positional encoding, and the roles of Encoder and Decoder components, using clear analogies and visual diagrams to aid comprehension.
Encoder-DecoderPositional EncodingSelf-Attention
0 likes · 20 min read

This comprehensive guide walks you through the Transformer architecture—from its encoder‑decoder structure and self‑attention mechanism to multi‑head attention, positional embeddings, and practical PyTorch implementations—providing clear visualizations and code examples for deep learning practitioners.
PyTorchSelf-AttentionTransformer
0 likes · 22 min read

This practical guide walks through the full Transformer architecture for Chinese‑to‑English translation, detailing encoder‑decoder structure, tokenization and embeddings, batch handling with padding and masks, positional encodings, parallel teacher‑forcing, self‑ and multi‑head attention, and the complete forward and back‑propagation training steps.
Positional EncodingPyTorchSelf-Attention
0 likes · 26 min read

This article walks readers through the fundamentals of large language models, covering what tokens are, how tokenization works, the conversion of tokens to numeric IDs, the transformer architecture—including positional encoding, self‑attention, feed‑forward networks and softmax—and explains how these components enable next‑token prediction.
EmbeddingLLMSelf-Attention
0 likes · 9 min read

This article explains the evolution of natural language processing, the limitations of rule‑based, statistical, and recurrent neural network models, and then introduces the Transformer architecture—covering word and position embeddings, self‑attention, multi‑head attention, Add & Norm, feed‑forward layers, and encoder‑decoder design—to help beginners grasp why Transformers solve key NLP problems.
AINLPSelf-Attention
0 likes · 15 min read

This article explains the historical challenges of natural language processing, from rule‑based and statistical models to recurrent networks and their limitations, then introduces the Transformer architecture, its self‑attention mechanism, multi‑head attention, and supporting layers, illustrating how it overcomes previous issues and enables efficient parallel training.
NLPSelf-AttentionTransformer
0 likes · 16 min read

This article explains the self‑attention mechanism behind large language models, detailing why static word importance fails, how queries, keys, and values are generated, how attention scores are computed, scaled, softmaxed, and used to produce context‑aware word vectors, while noting computational costs.
AILLMSelf-Attention
0 likes · 9 min read


AIWalkerJan 11, 2025 · Artificial Intelligence
CAS‑ViT introduces a convolutional additive self‑attention mechanism that dramatically reduces the computational cost of Vision Transformers, achieving state‑of‑the‑art accuracy on image classification, object detection, and segmentation while being deployable on mobile devices.
Computer VisionEfficient ModelsSelf-Attention
0 likes · 19 min read

This article explains what AI Transformers are, why they matter, their key components and mechanisms, various applications ranging from language processing to bioinformatics, and how they differ from traditional neural networks, providing a comprehensive overview of Transformer architecture and its impact on modern AI research.
AISelf-AttentionTransformer
0 likes · 20 min read


JavaEdgeJul 22, 2024 · Artificial Intelligence
This article explains the fundamentals of transformer models, why they outperform earlier neural networks, their core components such as self‑attention and positional encoding, practical use cases across language and biology, and how they differ from RNNs, CNNs, and other architectures.
AISelf-AttentionSequence-to-Sequence
0 likes · 20 min read

This article explains the fundamental architecture of large language models, from the dual file nature of parameters and code, through neural network basics, perceptrons, and weight training, to the Transformer’s tokenization, positional encoding, self‑attention, and inference processes, illustrated with diagrams and examples.
Neural NetworkSelf-AttentionTransformer
0 likes · 22 min read

This article explains the fundamental concepts behind large language models, including their two-file structure, training process, neural network basics, perceptron examples, weight and threshold calculations, the TensorFlow Playground, and a detailed walkthrough of the Transformer architecture with tokenization, positional encoding, self‑attention, normalization, and feed‑forward layers.
AISelf-AttentionTransformer
0 likes · 20 min read

This article provides a comprehensive, beginner-friendly explanation of the Transformer architecture, covering its encoder‑decoder structure, self‑attention, multi‑head attention, positional encoding, residual connections, decoding process, final linear and softmax layers, and training considerations, illustrated with numerous diagrams and code snippets.
Self-AttentionTransformerdeep learning
0 likes · 24 min read


JD TechJun 7, 2024 · Artificial Intelligence
This article explains the fundamentals of attention mechanisms, including biological inspiration, the evolution from early visual attention to modern self‑attention in Transformers, details the scaled dot‑product calculations, positional encoding, and multi‑head attention, illustrating how these concepts enable efficient parallel processing of sequence data.
AIAttentionPositional Encoding
0 likes · 12 min read


NewBeeNLPApr 26, 2024 · Artificial Intelligence
This article reviews the paper “Distinguished in Uniform: Self-Attention vs. Virtual Nodes,” comparing graph Transformers and MPGNNs with virtual nodes on theoretical consistency and experimental performance, revealing that neither approach universally dominates the other.
Graph Neural NetworksMPGNNSelf-Attention
0 likes · 9 min read


ArchitectMar 19, 2024 · Artificial Intelligence
This article explains the core principles of Transformer models—covering input embeddings, self‑attention, multi‑head attention, positional encoding, feed‑forward networks, and decoder strategies—using concrete examples like "The cat sat on the mat" and "The quick brown fox jumps over the lazy dog" to illustrate each step.
Encoder-DecoderFeed-Forward NetworkNLP
0 likes · 13 min read

This article explains the Transformer architecture, its self‑attention mechanism, encoder‑decoder design, and the profound impact it has had on natural language processing, computer vision, and large‑scale language models like GPT.
AI ArchitectureNLPSelf-Attention
0 likes · 6 min read

This article provides a comprehensive overview of the attention mechanism, its mathematical foundations, the transformer model architecture—including encoder and decoder components—and the BERT pre‑training model, detailing their principles, implementations, and applications in natural language processing.
Attention MechanismBERTEncoder-Decoder
0 likes · 13 min read


21CTOApr 27, 2023 · Artificial Intelligence
This article explains the Transformer model—from its encoder‑decoder structure and self‑attention mechanism to multi‑head attention, positional encoding, residual connections, training loss, and inference strategies—providing a clear, visual walkthrough for readers new to modern NLP architectures.
Self-AttentionTransformerdeep learning
0 likes · 21 min read

This article provides a concise technical overview of GPT, explaining how word vectors are constructed, how the Transformer architecture with self‑attention and feed‑forward layers processes these vectors, and how GPT‑2 and GPT‑3 extend the model with decoder‑only and large‑scale designs.
AIGPTSelf-Attention
0 likes · 8 min read

This article introduces the Transformer architecture, explains the self‑attention mechanism with visual illustrations, and provides a full, runnable PyTorch code example that implements the encoder‑decoder structure for sequence‑to‑sequence tasks.
NLPPyTorchSelf-Attention
0 likes · 11 min read

This article introduces the fundamentals of Vision Transformers (ViT) for computer‑vision developers, starting with an overview of the transformer architecture, detailed explanation of self‑attention and multi‑head attention, and step‑by‑step PyTorch code examples that illustrate query, key, value computation and attention scoring.
PyTorchSelf-AttentionTransformer
0 likes · 12 min read

Vivo’s frontend team built a self‑attention‑based machine‑learning model that predicts web‑page layout types (column, row, or absolute) from node dimensions and positions, solving parent‑child and sibling relationships for design‑to‑code conversion, achieving 99.4% accuracy using over 20 k labeled, crawled, and generated samples, while outlining further enhancements.
D2CNeural NetworkSelf-Attention
0 likes · 11 min read

This article explains how the Fine‑ and Coarse‑Granularity Hybrid Self‑Attention (FCA) mechanism reduces BERT’s computational cost by over 50% while keeping accuracy loss under 1%, detailing the method, experimental results, and its significance for efficient large‑scale language models.
BERTFCAModel Efficiency
0 likes · 8 min read

The paper proposes a robust Grad‑CAM‑inspired visualization for Vision Transformers that combines attention weights and gradients to generate class‑specific saliency maps, demonstrates superior alignment with discriminative regions across ViT, Swin and Volo models, and shows a 76% false‑positive reduction in Baidu’s porn‑content risk control system.
Grad-CAMInput VisualizationSelf-Attention
0 likes · 11 min read

An in‑depth Q&A breaks down core BERT concepts—from the purpose of the [CLS] token and masking strategies to self‑attention complexity, sparse attention tricks, subword handling of OOV words, warm‑up learning rates, GPT’s unidirectional nature, and ALBERT’s parameter sharing—providing concise explanations for each.
BERTSelf-AttentionSubword Tokenization
0 likes · 7 min read


Code DAODec 29, 2021 · Artificial Intelligence
The paper proposes a stand‑alone axial‑attention mechanism that converts 2‑D attention into 1‑D to lower computational cost while preserving global context, introduces position‑sensitive self‑attention, integrates it into Axial‑ResNet and Axial‑DeepLab, and demonstrates strong results on four large segmentation datasets.
Axial AttentionComputer VisionDeepLab
0 likes · 7 min read


Code DAODec 8, 2021 · Artificial Intelligence
This article walks through the design of Compact Transformers, explaining scaled dot‑product self‑attention, positional embeddings, multi‑head attention, and Vision Transformer architecture, and provides full PyTorch code so readers can train lightweight CV and NLP classifiers on a single PC.
Compact TransformersPatch EmbeddingPositional Embedding
0 likes · 19 min read


AntTechOct 29, 2021 · Artificial Intelligence
The Ant Insurance Technology team, together with the Institute of Automation of the Chinese Academy of Sciences, secured first place in both the MuSe‑Wilder and MuSe‑Sent tracks of the MuSe2021 Multimodal Sentiment Challenge held at the 29th ACM International Conference on Multimedia in Chengdu, showcasing advanced multimodal AI techniques.
BiLSTMMuSe2021Self-Attention
0 likes · 4 min read

This article explores how Transformer architectures, originally designed for NLP, are adapted for visual tasks, detailing pioneering models such as DETR, CBAM, NLNet, SENet, and GCNet, and explains their structures, attention mechanisms, advantages, and experimental findings for image processing.
DETRSelf-Attentionattention mechanisms
0 likes · 13 min read

This article explains the Transformer architecture—its self‑attention core, multi‑head attention, positional encoding, encoder‑decoder structure, and how it overcomes RNN limitations, providing a foundation for its use in NLP, image detection, and OCR.
NLPPositional EncodingSelf-Attention
0 likes · 7 min read

This article explains the BERT model by first reviewing the Encoder-Decoder framework, then detailing the attention mechanism—including self-attention and multi-head attention—before describing the full Transformer architecture and finally outlining BERT’s encoder-only design, training stages, and fine-tuning applications.
AttentionBERTEncoder-Decoder
0 likes · 15 min read

This article provides a comprehensive, illustrated walkthrough of OpenAI's GPT‑2 language model, covering its decoder‑only Transformer architecture, self‑attention mechanisms, token processing, training data, differences from BERT, and applications beyond language modeling, enriched with visual diagrams and code snippets for deeper understanding.
AIGPT-2Language Model
0 likes · 24 min read

This article provides a step‑by‑step illustrated guide to the Transformer architecture, covering its macro structure, detailed self‑attention mechanisms, multi‑head attention, positional encoding, residual connections, decoder operation, training process, loss functions, and includes complete PyTorch and custom Python code examples.
NLPPyTorchSelf-Attention
0 likes · 33 min read

This article presents DeepMatch, a behavior‑sequence based deep learning recall model enhanced with play‑rate and intent‑type embeddings, describes its self‑attention architecture, factorized embedding parameterization, multitask loss design, distributed TensorFlow training tricks, and demonstrates significant offline and online improvements in music recommendation performance.
Self-AttentionTensorFlowdeep learning
0 likes · 15 min read

This article examines the DeepMatch retrieval model for Tmall Genie music recommendation, detailing how incorporating user feedback such as play‑rate and query intent signals via multi‑task learning and feedback‑aware self‑attention improves recall accuracy and reduces negative recommendations, while also discussing embedding factorization, loss functions, and distributed training optimizations.
Self-Attentiondeep learningmultitask learning
0 likes · 18 min read

This article details Alibaba’s Taobao search CTR/CVR user modeling approach, covering background, model architecture with self‑attention and attention pooling, handling short‑term, long‑term, and on‑device behavior sequences, experimental results showing AUC improvements, and future directions.
CTR predictionSelf-AttentionUser Modeling
0 likes · 20 min read

This article systematically reviews the history, core concepts, variants, and practical implementations of attention mechanisms—from early additive and multiplicative forms to self‑attention, multi‑head attention, and recent transformer‑based models—highlighting why attention has become fundamental in modern AI research.
AttentionNLPSelf-Attention
0 likes · 16 min read

This article introduces GATNE, a graph embedding framework that jointly models heterogeneous nodes, multiple edge types, and rich node attributes using base and edge embeddings, self‑attention, and inductive learning, and demonstrates its superior performance on several real‑world datasets.
GATNESelf-Attentiongraph embedding
0 likes · 8 min read

This article systematically reviews the evolution of attention mechanisms—from early additive and multiplicative forms to self‑attention and multi‑head variants—explaining their core three‑step framework, key differences, and why they have become essential across NLP, vision, and broader AI applications.
NLPSelf-AttentionTransformer
0 likes · 19 min read

This article provides a comprehensive, step‑by‑step explanation of the Transformer architecture, covering its encoder‑decoder structure, self‑attention, multi‑head attention, positional encoding, residual connections, and training processes, illustrated with diagrams and code snippets to aid readers new to neural machine translation.
Neural Machine TranslationPositional EncodingSelf-Attention
0 likes · 16 min read

This paper proposes a novel attention‑based framework that groups and encodes heterogeneous user behavior sequences into separate semantic subspaces, applies self‑attention to capture inter‑behavior influences, and demonstrates faster training and comparable or improved recommendation performance across multiple tasks and datasets.
Self-AttentionUser Modelingheterogeneous behavior
0 likes · 12 min read
