Tagged articles
multi-head attention
18 articles · Page 1 of 1
The paper proposes CAIM, a context-aware user intent modeling framework for e-commerce relevant recommendation that decomposes intent into category, brand, shop, and price sub-intents via three modules (IEM, IRM, SIEM), achieving 0.20–0.21 AUC gains offline and 6.9% order and 7.7% GMV lifts in a 14-day online A/B test.
A/B testingAUCCIKM 2026
0 likes · 13 min read

The article explains how the Transformer architecture replaces RNNs with self‑attention, detailing the Q‑K‑V mechanism, positional encodings such as RoPE, multi‑head attention, modern improvements like SwiGLU and RMSNorm, and provides formulas for parameter and FLOP estimation.
FFNPositional EncodingRMSNorm
0 likes · 26 min read

The article explains why modern interviewers ask about Transformer fundamentals, breaks down its core components such as self‑attention, multi‑head attention, feed‑forward networks, residual connections and positional encodings, and demonstrates a complete PyTorch toy model that predicts the sum‑mod‑10 of integer sequences while visualizing loss curves, attention heatmaps, embedding PCA and early‑stage gradient norms.
Gradient AnalysisModel VisualizationPyTorch
0 likes · 20 min read


CodePathJun 3, 2026 · Artificial Intelligence
The article dissects how the 2017 "Attention Is All You Need" paper sparked a fundamental redesign of sequence modeling by replacing recurrent and convolutional approaches with self‑attention, detailing its mathematical foundations, architectural components, training tricks, limitations, and emerging alternatives such as Mamba.
Attention MechanismMambaPositional Encoding
0 likes · 24 min read

This article walks through MiniMind's Attention.forward implementation, explaining why Q, K, and V are created, how tensors are reshaped for multi‑head attention, the role of masks, KV cache, GQA, and how each token aggregates information from the entire context.
AttentionGQAKV cache
0 likes · 21 min read

This article provides a comprehensive, step‑by‑step walkthrough of the Transformer architecture, covering input embedding, positional encoding, the mechanics of Q‑K‑V attention, scaled dot‑product formulas, multi‑head and masked attention, feed‑forward networks, residual connections, layer normalization, decoder generation, and recent attention‑optimization techniques.
AttentionFeed-Forward NetworkPositional Encoding
0 likes · 39 min read

This article breaks down the core components of the Transformer architecture—including input embedding, positional encoding, multi‑head self‑attention, residual connections with layer normalization, position‑wise feed‑forward networks, and the rationale behind stacking multiple encoder layers—using clear explanations and illustrative diagrams.
Add&NormFeed ForwardInput Embedding
0 likes · 12 min read

This article explains the Transformer architecture from "Attention Is All You Need", covering self-attention, multi-head attention, layer normalization, positional encoding, and the encoder-decoder structure with PyTorch code, showing how Transformers enable parallel sequence processing without recurrence.
Attention Is All You NeedEncoder-DecoderLayer Normalization
0 likes · 10 min read

This guide breaks down the core logic, computation steps, formulas, and common interview questions about Self‑Attention and Multi‑Head Attention in Transformers, offering concrete explanations, dimensional examples, and practical answering techniques to help candidates ace large‑model algorithm interviews.
Interview TipsSelf-AttentionTransformer
0 likes · 8 min read

This article demystifies the Transformer architecture for beginners, explaining its relationship to large models, the self‑attention and multi‑head attention mechanisms, positional encoding, and the roles of Encoder and Decoder components, using clear analogies and visual diagrams to aid comprehension.
Encoder-DecoderPositional EncodingSelf-Attention
0 likes · 20 min read

The paper introduces MHA2MLA, a data‑efficient fine‑tuning framework that converts pre‑trained multi‑head attention LLMs to DeepSeek’s Multi‑Head Latent Attention architecture, achieving up to 92% KV‑cache compression with less than 0.5% performance loss on long‑context tasks.
LLMLow-Rank ApproximationMulti-Head Latent Attention
0 likes · 8 min read

The article delivers a PPT‑style deep dive into the DeepSeek series—from the original LLM through DeepSeek‑MoE, Math, V2, V3 and R1—highlighting core innovations such as Multi‑Head Latent Attention, fine‑grained MoE, GRPO reinforcement learning, Multi‑Token Prediction, DualPipe parallelism and FP8 training that together achieve high performance at a fraction of traditional costs, and notes their integration into Tencent’s OlaChat intelligent assistant.
AIDeepSeekFP8 training
0 likes · 21 min read


ArchitectMar 19, 2024 · Artificial Intelligence
This article explains the core principles of Transformer models—covering input embeddings, self‑attention, multi‑head attention, positional encoding, feed‑forward networks, and decoder strategies—using concrete examples like "The cat sat on the mat" and "The quick brown fox jumps over the lazy dog" to illustrate each step.
Encoder-DecoderFeed-Forward NetworkNLP
0 likes · 13 min read


Code DAODec 8, 2021 · Artificial Intelligence
This article walks through the design of Compact Transformers, explaining scaled dot‑product self‑attention, positional embeddings, multi‑head attention, and Vision Transformer architecture, and provides full PyTorch code so readers can train lightweight CV and NLP classifiers on a single PC.
Compact TransformersPatch EmbeddingPositional Embedding
0 likes · 19 min read

DeepXi introduces a two‑stage deep learning framework for speech enhancement, using prior SNR estimation and MMSE gain, while the MHANet extension leverages multi‑head attention to model long‑range dependencies, with detailed training strategies, model compression to GRU, deployment via TFLite, and impressive low‑latency results.
GRUTFLiteaudio denoising
0 likes · 8 min read

This article explains the Transformer architecture—its self‑attention core, multi‑head attention, positional encoding, encoder‑decoder structure, and how it overcomes RNN limitations, providing a foundation for its use in NLP, image detection, and OCR.
NLPPositional EncodingSelf-Attention
0 likes · 7 min read

This article provides a step‑by‑step illustrated guide to the Transformer architecture, covering its macro structure, detailed self‑attention mechanisms, multi‑head attention, positional encoding, residual connections, decoder operation, training process, loss functions, and includes complete PyTorch and custom Python code examples.
NLPPyTorchSelf-Attention
0 likes · 33 min read

This article provides a comprehensive, step‑by‑step explanation of the Transformer architecture, covering its encoder‑decoder structure, self‑attention, multi‑head attention, positional encoding, residual connections, and training processes, illustrated with diagrams and code snippets to aid readers new to neural machine translation.
Neural Machine TranslationPositional EncodingSelf-Attention
0 likes · 16 min read
