Tagged articles
FlashAttention
19 articles · Page 1 of 1
This article dissects MHA, MQA, GQA, and FlashAttention across five analytical layers, revealing how structural optimizations (MQA/GQA) reduce KV Cache memory by sharing keys/values, while FlashAttention tackles memory bandwidth via tiled computation and online softmax, and why modern LLMs combine GQA with FlashAttention for 2-4x speedups without accuracy loss.
Attention MechanismFlashAttentionGQA
0 likes · 11 min read

This comprehensive survey covers attention mechanisms from 2017's Scaled Dot-Product Attention to 2026's NSA and FlashAttention-4, organizing them into four optimization tracks—KV cache compression, exact attention acceleration, long-context extension, and frontier alternative architectures—with concrete benchmarks, trade-offs, and practical selection criteria for each use case.
FlashAttentionHybrid ArchitecturesKV Cache
0 likes · 12 min read

This article analyzes why LLM inference often yields nondeterministic results, explains how floating‑point addition order, GEMM tiling, Split‑K, RMSNorm, FlashAttention, and NCCL all contribute to batch variance, and details the engineering steps vLLM takes to enforce batch‑invariant execution across GPUs.
Batch InvarianceDeterminismFlashAttention
0 likes · 52 min read

The article analyzes recent AI updates—from DeepSeek's DSpark inference boost and FlashAttention‑4's kernel redesign to Codex UI tweaks and design‑mode tools—arguing that the competition is shifting from answering questions to actually completing tasks, and it highlights three layers of progress, evaluation challenges, and the practical questions we must now ask of AI agents.
AIAgent AutomationDeepSeek
0 likes · 19 min read

Although a KV‑cache eviction strategy can discard 90% of tokens, GPU memory usage stays almost unchanged because paged‑attention memory blocks remain occupied and fast attention kernels discard the full score matrix, preventing effective memory release.
FlashAttentionGPU memoryKV Cache
0 likes · 7 min read

This article walks through a two‑month source‑code study of vLLM, explaining how token‑level scheduling, continuous batching, and the Paged Attention mechanism reshape tensor dimensions to turn large‑model inference into a compute‑bound, high‑throughput process while managing GPU memory efficiently.
FlashAttentionGPU optimizationLLM inference
0 likes · 29 min read

The vLLM 0.20.0 release dramatically upgrades the inference engine with DeepSeek V4 support, default CUDA 13, PyTorch 2.11, Transformers v5 compatibility, FlashAttention 4 MLA prefill, TurboQuant 2‑bit KV cache, an online quantization front‑end, IR enhancements, Model Runner V2 features, and a slew of new models, while providing detailed installation and upgrade guidance.
CUDA 13FlashAttentionTransformers
0 likes · 10 min read

This article examines the Mixture‑of‑Depths Attention (MoDA) mechanism, detailing its novel flash‑compatible KV layout, combined sequence‑depth attention, theoretical analysis, and extensive experiments that show significant reductions in validation loss and accuracy gains on downstream tasks compared to the OLMo2 baseline.
Attention MechanismDeep KVFlashAttention
0 likes · 9 min read

This article explains why TPU programming differs from GPU, describes the explicit HBM‑VMEM‑register data movement required on TPU, introduces the Pallas grid‑BlockSpec‑Ref model, and walks through four progressively more complex kernels—including element‑wise add, tiled dot product, fused RMSNorm with scratch memory, and a production‑grade FlashAttention implementation—showing how each kernel maps to the TPU memory hierarchy and leverages Pallas features such as input_output_aliases and PrefetchScalarGridSpec.
FlashAttentionJAXPallas
0 likes · 20 min read

The vLLM 0.17.0 release brings FlashAttention 4 integration, a mature Model Runner V2, complete Qwen 3.5 series support, a one‑click performance‑mode flag, Anthropic API compatibility, advanced weight‑offloading, broader hardware support beyond NVIDIA, ASR model integration, and detailed upgrade and installation guidance.
ASRAnthropic APIFlashAttention
0 likes · 12 min read

This tutorial walks readers through the fundamentals of YOLOv12, covering model variants, dataset preparation with Roboflow, optional FlashAttention acceleration, installation, model selection, training commands, post‑training tasks such as tracking, validation, inference, exporting to ONNX, and benchmarking, all with concrete code snippets and practical tips.
Computer VisionFlashAttentionModel Training
0 likes · 8 min read

FlashAttention accelerates transformer inference by combining nine techniques—including loss‑less attention, GPU memory‑pyramid optimization, SRAM‑reusing tiling, safe softmax scaling, online buffering, tile‑size constraints, parallel multiplication, reduced KV slicing, and integrated backward‑pass caching—to achieve efficient, high‑throughput computation on modern GPUs.
Attention MechanismFlashAttentionGPU optimization
0 likes · 8 min read

The Bilibili TTV team optimized OpenSora and CogVideoX text‑to‑video models by redesigning data storage with Alluxio, parallelizing VAE encoding, applying dynamic sequence‑parallel and DeepSpeed‑Ulysses attention, adapting GPU code for NPU execution, leveraging profiling‑driven kernel fusion, FlashAttention, and expandable memory to dramatically increase training efficiency and frame throughput, while outlining future pipeline‑parallel and ZeRO‑3 scaling plans.
FlashAttentionNPUdata pipeline
0 likes · 26 min read

This article explains the FlashAttention algorithm, its memory‑efficient tiling and recomputation techniques, and how enabling the flash_attn flag dramatically speeds up Qwen‑series large models while outlining hardware, software requirements and potential trade‑offs.
AttentionFlashAttentionGPU optimization
0 likes · 8 min read


NewBeeNLPNov 18, 2024 · Artificial Intelligence
This article examines various techniques for compressing and accelerating the KV cache in transformer models—including MQA, GQA, MLA, sliding‑window and linear attention, flash attention, page and ring attention, as well as mixed‑precision training and ZeRO parallelism—providing code snippets, implementation details, and practical trade‑offs.
AttentionFlashAttentionKV Cache
0 likes · 17 min read

This article presents a concise, step‑by‑step derivation of FlashAttention, explaining the prerequisite linear‑algebra concepts, the softmax simplifications, and the parallel computation workflow—including the LSE‑enhanced version—so readers can grasp the algorithm’s elegance without heavy mathematics.
Algorithm DerivationAttention MechanismFlashAttention
0 likes · 8 min read

This article explains the core mechanisms of Transformer models, details the Rotational Position Embedding (RoPE) and FlashAttention techniques for handling long sequences, introduces the GLM-4-Plus series, and presents an empirical evaluation on the THUCNews dataset showing its superior long-text performance.
FlashAttentionGLM-4-PlusLong Text
0 likes · 13 min read

Deploying a dedicated inference cluster and applying four key optimizations—FlashAttention‑based attention computation, PageAttention KV‑cache management, Mixture‑of‑Experts parameter reduction, and tensor parallelism—can accelerate large language model inference by up to 50% for models as large as 70 B parameters while cutting deployment costs.
FlashAttentionPageAttentioninference acceleration
0 likes · 17 min read

During the Saturday session, we reviewed ChatGLM2’s upgrades—Multi‑Query Attention and FlashAttention—demonstrated deployment on Ascend + ModelArts + MindSpore, and introduced ChatGLM3’s revamped prompt design, native tool‑calling and code‑interpreter capabilities, while previewing the next lecture on text‑generation decoding.
ChatGLM2ChatGLM3FlashAttention
0 likes · 6 min read
