Tagged articles
Positional Encoding
22 articles · Page 1 of 1
The article explains why vanilla Transformers cannot sense word order, introduces sinusoidal positional encoding as the elegant solution, compares it with learned and relative encodings, and shows how modern variants like RoPE preserve both absolute and relative positions for long texts.
Learned EncodingPositional EncodingRelative Position
0 likes · 9 min read

The article explains how the Transformer architecture replaces RNNs with self‑attention, detailing the Q‑K‑V mechanism, positional encodings such as RoPE, multi‑head attention, modern improvements like SwiGLU and RMSNorm, and provides formulas for parameter and FLOP estimation.
FFNPositional EncodingRMSNorm
0 likes · 26 min read

This article explains how modern AI large models—such as GPT‑4, Claude 3.5, Gemini, Llama, and DeepSeek—rely on the Transformer architecture, detailing token embeddings, self‑attention, positional encoding, and the three‑stage training pipeline of pre‑training, instruction fine‑tuning, and alignment optimization.
Model TrainingPositional EncodingSelf-Attention
0 likes · 5 min read


CodePathJun 3, 2026 · Artificial Intelligence
The article dissects how the 2017 "Attention Is All You Need" paper sparked a fundamental redesign of sequence modeling by replacing recurrent and convolutional approaches with self‑attention, detailing its mathematical foundations, architectural components, training tricks, limitations, and emerging alternatives such as Mamba.
Attention MechanismMambaPositional Encoding
0 likes · 24 min read

This article provides a comprehensive, step‑by‑step walkthrough of the Transformer architecture, covering input embedding, positional encoding, the mechanics of Q‑K‑V attention, scaled dot‑product formulas, multi‑head and masked attention, feed‑forward networks, residual connections, layer normalization, decoder generation, and recent attention‑optimization techniques.
AttentionFeed-Forward NetworkPositional Encoding
0 likes · 39 min read

This article breaks down the core components of the Transformer architecture—including input embedding, positional encoding, multi‑head self‑attention, residual connections with layer normalization, position‑wise feed‑forward networks, and the rationale behind stacking multiple encoder layers—using clear explanations and illustrative diagrams.
Add&NormFeed ForwardInput Embedding
0 likes · 12 min read

This article explains the core principles of Neural Radiance Fields, detailing how a fully‑connected network maps 5‑D coordinates to color and density, the role of positional encoding and hierarchical sampling, and provides a complete PyTorch implementation with training and rendering examples.
3D Scene RepresentationHierarchical SamplingNeRF
0 likes · 18 min read

This article explains the Transformer architecture from "Attention Is All You Need", covering self-attention, multi-head attention, layer normalization, positional encoding, and the encoder-decoder structure with PyTorch code, showing how Transformers enable parallel sequence processing without recurrence.
Attention Is All You NeedEncoder-DecoderLayer Normalization
0 likes · 10 min read

This article deeply explores the Transformer architecture by combining original paper principles with PyTorch source code, covering encoder‑decoder design, positional encoding assumptions, core parameters, residual connections, attention mechanisms, and detailed implementation snippets to help readers understand and reproduce the model.
AttentionPositional EncodingPyTorch
0 likes · 22 min read

Preparing for large-model interview? This guide reveals why interviewers probe seemingly minor components—positional encoding, residual connections, layer normalization, and feed-forward networks—explains each technique's purpose, variants, and how to answer confidently, plus practical tips and a learning roadmap to boost your chances.
FFNInterview TipsLayerNorm
0 likes · 8 min read

This article demystifies the Transformer architecture for beginners, explaining its relationship to large models, the self‑attention and multi‑head attention mechanisms, positional encoding, and the roles of Encoder and Decoder components, using clear analogies and visual diagrams to aid comprehension.
Encoder-DecoderPositional EncodingSelf-Attention
0 likes · 20 min read

This practical guide walks through the full Transformer architecture for Chinese‑to‑English translation, detailing encoder‑decoder structure, tokenization and embeddings, batch handling with padding and masks, positional encodings, parallel teacher‑forcing, self‑ and multi‑head attention, and the complete forward and back‑propagation training steps.
Positional EncodingPyTorchSelf-Attention
0 likes · 26 min read


AntTechMar 4, 2025 · Artificial Intelligence
This article introduces GraphCLIP, a self‑supervised graph‑summary pre‑training framework that boosts zero‑ and few‑shot transferability of graph foundation models for text‑attributed graphs, and 2D‑TPE, a two‑dimensional positional encoding method that preserves table structure to markedly improve large language model performance on table‑understanding tasks, while also announcing a live paper session at WWW 2025 featuring the authors.
Graph Neural NetworksPositional EncodingSelf‑Supervised Learning
0 likes · 6 min read


NewBeeNLPNov 27, 2024 · Artificial Intelligence
This article reviews the principles of absolute and relative positional encodings, explains why window extrapolation is crucial for large language models, analyzes current extrapolation methods, evaluates their performance, and answers common questions about extending LLM context windows.
LLMPositional EncodingRoPE
0 likes · 14 min read


JD TechJun 7, 2024 · Artificial Intelligence
This article explains the fundamentals of attention mechanisms, including biological inspiration, the evolution from early visual attention to modern self‑attention in Transformers, details the scaled dot‑product calculations, positional encoding, and multi‑head attention, illustrating how these concepts enable efficient parallel processing of sequence data.
AIAttentionPositional Encoding
0 likes · 12 min read


ArchitectMar 19, 2024 · Artificial Intelligence
This article explains the core principles of Transformer models—covering input embeddings, self‑attention, multi‑head attention, positional encoding, feed‑forward networks, and decoder strategies—using concrete examples like "The cat sat on the mat" and "The quick brown fox jumps over the lazy dog" to illustrate each step.
Encoder-DecoderFeed-Forward NetworkNLP
0 likes · 13 min read

This article analyzes upgrade paths of major LLMs—ChatGLM, LLaMA, Baichuan—detailing performance, context length, and architectural changes, then examines essential capabilities, data cleaning, tokenizer and attention design, and offers practical guidance for balanced scaling and efficient model construction.
BaichuanChatGLMLLM Architecture
0 likes · 32 min read

This article explains how to build the input processing stage of a Transformer—including tokenization with Hugging Face tokenizers, token‑to‑embedding conversion using BERT models, custom BPE tokenizers, and positional encoding—providing complete Python code examples and test results.
BPEEmbeddingPositional Encoding
0 likes · 14 min read


Code DAOMay 3, 2022 · Artificial Intelligence
This tutorial walks through the theory and implementation of Neural Radiance Fields (NeRF) in PyTorch, covering positional encoding, the MLP architecture, differentiable volume rendering, hierarchical sampling, training tricks, and references to the original research.
Hierarchical SamplingNeRFNeural Radiance Fields
0 likes · 23 min read

This article revisits the long‑standing question of why BERT’s token, segment, and position embeddings are summed, critiques earlier explanations, and presents findings from the ICLR‑2021 paper “Rethinking Positional Encoding in Language Pre‑training” that show removing the token‑position cross term speeds convergence and improves downstream GLUE scores.
BERTEmbeddingLanguage Pretraining
0 likes · 6 min read

This article explains the Transformer architecture—its self‑attention core, multi‑head attention, positional encoding, encoder‑decoder structure, and how it overcomes RNN limitations, providing a foundation for its use in NLP, image detection, and OCR.
NLPPositional EncodingSelf-Attention
0 likes · 7 min read

This article provides a comprehensive, step‑by‑step explanation of the Transformer architecture, covering its encoder‑decoder structure, self‑attention, multi‑head attention, positional encoding, residual connections, and training processes, illustrated with diagrams and code snippets to aid readers new to neural machine translation.
Neural Machine TranslationPositional EncodingSelf-Attention
0 likes · 16 min read
