Tagged articles
sparse attention
75 articles · Page 1 of 1
Prefrontal AI and ETH Zurich introduce Latent Sparse Attention (LSA), which clusters keys in a learned latent space and uses hierarchical bucket search to reduce indexer complexity from quadratic to O(T log T), achieving 22.7x prefill speedup over GLM-5.3-Flash and 72.4x over Qwen3.8-27B at 10M tokens while maintaining accuracy.
DeepSeekGLM-5.3-FlashLLM inference
0 likes · 18 min read

Alibaba's Qwen4-Max uses a 2.4T-parameter MoE architecture activating only 71B parameters per token, achieving 58% lower per-task costs with native vision and 1M context, but the flagship model remains closed-source while smaller open-weight versions are promised later.
AI benchmarksAlibaba CloudLLM pricing
0 likes · 11 min read

A Japanese developer implemented Qwen Sparse Attention (QSA) on Qwen3.8-27B, achieving 2.92x faster prefill and 1.79x faster decode at 261K tokens while maintaining perplexity and DocVQA scores; a quantized version further reaches 2375 tok/s prefill and 115 tok/s decode with 22.8 GiB VRAM.
Community DevelopmentInference OptimizationQSA
0 likes · 5 min read

Baidu open-sources ERNIE-5.5-Flash-Base, a 280B-parameter sparse MoE model activating only 14B parameters per token, trained on 33.5T tokens with 128K context, achieving same-size SOTA against DeepSeek-V4-Flash-Base and others through hybrid sparse attention, multi-router latent MoE, and ECHO future-token prediction.
BaiduECHOERNIE-5.5-Flash-Base
0 likes · 19 min read

This article traces the evolution of attention mechanisms from early visual attention and RNN-based attention through the Transformer revolution to modern sparse, linear, and state-space variants, detailing recent innovations like GQA, MLA, dynamic sparse attention (MoBA, NSA), hybrid architectures, and cross-layer memory sharing in open-source LLMs such as Llama, DeepSeek, and Qwen.
Attention MechanismCross-Layer SharingDynamic Sparse Attention
0 likes · 17 min read

Community clues suggest Alibaba's Qwen 4 is near release, with inference optimizations in vLLM and llama.cpp, architectural details like a 27B dense model with 5120 hidden dim, 24 query heads, 4 KV heads, 4 residual streams, MTP, and QSA sparse attention, while anonymous Arena models Kalmia and Kerria are speculated as Qwen 4 variants showing strong code and 3D generation.
AlibabaArena benchmarkFP8 quantization
0 likes · 7 min read

Baidu open-sources Unlimited OCR 2, a 3B-parameter document parsing model with only 500M activated parameters, achieving state-of-the-art on MarathonDocBench (77.06) and MPDocBench (84.59), processing 5000+ page PDFs in one pass with 96% KV memory reduction via SR-SWA attention and unlimited decoding.
BaiduLong Document UnderstandingOCR
0 likes · 15 min read

PixVerse R2 demonstrates real-time world models that generate video and audio synchronously while accepting live inputs, but faces hurdles in compute costs, long-run stability, and distribution models for interactive content.
Generative AIPixVerse R2autoregressive generation
0 likes · 11 min read

This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.
DeepSeekGRPOKV cache optimization
0 likes · 37 min read

DeepSeek open-sources six low-level libraries for Huawei Ascend 950, covering operator DSL, matrix multiplication, MoE communication, sparse attention, and TopK selection, with reported near-peak hardware performance but requiring version-specific integration.
AI kernelsDeepEPDeepGEMM
0 likes · 14 min read

Naive AI demonstrates AI-driven R&D where agents autonomously redesign model architectures, optimize training systems, and achieve breakthrough benchmarks—Naive-N0.5-Flash surpasses larger models on coding and research tasks, while AI-built NaiveRT delivers 30-50x inference speedups and AutoWM sets new world model records through self-discovered algorithms.
AI for AIAI research agentsInference Optimization
0 likes · 22 min read

An Oxford PhD thesis introduces three methods—VCD, COMET, and SPARTAN—that embed causal inductive biases into world models, achieving better adaptability, robustness, and interpretability across simulation, reinforcement learning, and language modeling tasks.
COMETInterpretabilitySPARTAN
0 likes · 19 min read

Researchers from Peking University, Tsinghua University, and Alibaba introduce SparkDiffusion, a unified framework combining sparse attention, few-step distillation, and FP8 quantization to achieve 265× acceleration for DiT video generation on a single RTX 5090, overcoming the high-sparsity trap that degrades quality at extreme sparsity.
DiTFP8 quantizationacceleration
0 likes · 12 min read

This article traces the evolution of attention mechanisms in large language models, showing how hybrid architectures now combine linear and sparse attention — exemplified by GLM-5.3-Flash integrating Kimi's KDA and DeepSeek's DSA — driven by shifting constraints from context length to agent workloads, with MiniMax's architectural journey illustrating the trade-offs.
Context ScalingDeepSeekKimi
0 likes · 17 min read

Huawei Cloud adapts and optimizes the 280B‑parameter dots3‑note preview model released by Xiaohongshu, detailing its mixed sparse attention, MoE enhancements, multimodal pre‑training pipelines, unified reward training, and a suite of inference‑time optimizations—including FlashComm, FUSED_MC2, and MTP speculative decoding—delivered via the vLLM Ascend framework for high‑throughput, low‑latency deployment.
Huawei CloudMTP Speculative DecodingMoE optimization
0 likes · 7 min read


21CTOAug 11, 2026 · Artificial Intelligence
The article details how antirez rebuilt a zero‑dependency, high‑throughput MiniMax‑H3 video/audio inference engine entirely in C for Apple Silicon, achieving 22‑frame 512×512 video rendering in about 3.5 seconds on an M5 Max, and explains the engineering tricks, benchmark results, adjustable quality settings, terminal UI, community feedback, and upcoming sparse‑attention support.
Apple SiliconC++INT8 quantization
0 likes · 7 min read

HiLS-Attention introduces a hierarchical landmark sparse attention that mathematically resolves chunk‑importance estimation and end‑to‑end differentiability, matching full‑attention perplexity while delivering 13.5× faster prefill, 15.7× faster decode, and up to 512‑fold context extrapolation, even surpassing full attention on long‑context retrieval tasks.
EfficiencyHiLS-AttentionTransformer
0 likes · 11 min read

Meituan has open‑sourced LongCat-2.0, a 1.6‑trillion‑parameter model that runs inference on domestic GPU/NPUs through sparse attention, N‑gram embedding, and a MOPD architecture, providing multi‑precision weights and deployment scripts to enable stable, high‑throughput serving on existing Chinese hardware.
Inference OptimizationLongCat-2.0N-gram embedding
0 likes · 7 min read

Meituan’s LongCat‑2.0, a 1.6‑trillion‑parameter MoE model trained on a 50,000‑card domestic cluster, supports 1 M‑token context, uses Sparse Attention, zero‑compute experts and MOPD architecture, achieving over 1 T tokens/day throughput, 1.5× MFU efficiency, and top‑ranked scores on coding and agent benchmarks.
AI ModelAgentic CodingLongCat-2.0
0 likes · 10 min read

The article examines how 1‑million‑token context has become a standard feature in modern LLMs, explains the compute and memory challenges it brings, reviews the sparse‑attention and token‑selection tricks (including GLM 5.2’s IndexShare and LayerSplit), and outlines practical evaluation methods for measuring long‑context effectiveness.
1M contextGLM-5.2IndexShare
0 likes · 10 min read

The article analyzes GLM‑5.2’s surprising performance—ranking first in front‑end web design, surpassing Claude Opus on Code Arena, and leveraging DeepSeek‑derived sparse attention and IndexShare optimizations—while noting its weaker long‑context engineering scores and highlighting hardware scarcity as the main bottleneck for Chinese LLMs.
AI EfficiencyDeepSeekGLM-5.2
0 likes · 11 min read

In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.
4-bit quantizationKV compressionMulti-Token Prediction
0 likes · 13 min read

MiniMax Sparse Attention (MSA) is an open‑source library that delivers high‑performance dense and block‑sparse attention operators for NVIDIA SM100 GPUs by combining a Jinja‑based csrc JIT stack with a Cutlass Python DSL (CuTe‑DSL), enabling low‑precision quantization, paging support, and seamless migration from dense code.
AI kernelsCuTe-DSLCutlass
0 likes · 5 min read

Keye‑VL‑2.0, an open‑source MoE multimodal foundation model, tackles hour‑level video understanding and agentic intelligence by embedding DeepSeek Sparse Attention into a GQA‑based architecture, enabling near‑lossless 256 K token context, four‑stage pre‑training, diverse RL distillation techniques, and achieving state‑of‑the‑art results on long‑video benchmarks, with weights publicly released.
MoEPretrainingRL distillation
0 likes · 8 min read

The article introduces Tencent Hunyuan's Stem sparse‑attention algorithm, which reduces first‑token latency by 3.6× on 128K context LLMs by reallocating compute with Token Position Decay and Output‑Aware Metric, and validates the gains with HPC‑optimized operators that outperform existing sparse methods in extensive benchmarks.
HPC OperatorsLLM inferenceOutput-Aware Metric
0 likes · 11 min read

The MiniMax M3 model combines a 1‑million‑token context window, native multimodal training and a new MiniMax Sparse Attention architecture that cuts token compute to one‑twentieth of its predecessor, achieving up to 15× faster decoding, while its interactive user‑simulator training enables fully autonomous agents that can reproduce ICLR‑2025 research and tackle Auto‑Kaggle competitions at a fraction of the cost of Western models.
Agentic AIAuto KaggleM3
0 likes · 9 min read

MiniMax M3 claims to be the first open‑source LLM that simultaneously delivers top‑tier coding/agentic ability, a 1‑million‑token context window, and native multimodal understanding, and our benchmarks on coding suites, long‑context efficiency, and multimodal tasks confirm it exceeds expectations.
1M contextMiniMax M3Open Source LLM
0 likes · 15 min read


SuanNiJun 1, 2026 · Artificial Intelligence
MiniMax M3, a domestically developed LLM, combines a new sparse‑attention MSA architecture, native multimodal support, and million‑token context to match or surpass top closed‑source models in programming and agent benchmarks, while achieving a 9.4× speedup on FP8 GEMM and preparing for open‑source release.
AIFP8 GEMMMiniMax M3
0 likes · 12 min read

The article presents RTPurbo, a lightweight two‑stage training method that converts full‑attention LLMs into highly sparse models with over 97% sparsity, achieving up to 9.36× prefill and 2.01× decode speedups while preserving near‑lossless accuracy across long‑context benchmarks up to 512K tokens.
Dynamic Token SelectionLLM inferenceLow-Dimensional Projection
0 likes · 17 min read

This article presents GQLA, a single‑author variant of MLA that eliminates three hardware‑related drawbacks of MLA, demonstrates how it achieves balanced compute‑memory performance on both high‑end H100 and more modest H20 GPUs, and details conversion methods (TransGQLA) and sparse extensions with concrete benchmark results.
AttentionGQLALLM
0 likes · 16 min read


SuanNiMay 27, 2026 · Artificial Intelligence
The article reveals that xAI’s upcoming Grok-5 (Grok V9-Medium) will feature a 1.5-trillion-parameter model trained with extensive Cursor programming data, while MiniMax-M3 introduces a new sparse-attention architecture that boosts pre-fill speed by 9.7× and decode speed by 15.6×, highlighting a strategic partnership between SpaceX, Cursor, and xAI.
AI modelsCursorGrok-5
0 likes · 5 min read

SORT re‑architects industrial‑scale ranking by shifting to a request‑centric data paradigm, integrating sparse and MoE optimizations into a Transformer backbone, and delivering significant CTR‑AUC, FLOPs, and online metric improvements while maintaining high training and inference efficiency.
Large Model ScalingRankingTransformer
0 likes · 19 min read

The article analyzes SubQ, a new LLM architecture using Subquadratic Sparse Attention (SSA) to achieve a 12‑million‑token context window with linear compute scaling, delivering up to 52× speedup and costing just 5% of Opus while matching dense‑attention performance on long‑context benchmarks.
SSASubQbenchmark
0 likes · 14 min read

The Chinese Academy of Sciences unveils SpikingBrain 2.0‑5B, a brain‑inspired large model that uses dual‑space sparse attention and dual activation (FP8 and INT8‑Spiking) to cut training cost by over tenfold, achieve up to 15× speedup on long sequences, and match Qwen‑3 performance while drastically reducing power consumption.
SpikingBrain2.0benchmark performancebrain-inspired AI
0 likes · 10 min read

The SubQ model introduces Subquadratic Sparse Attention (SSA), a content‑dependent routing mechanism that reduces attention complexity to linear, enabling a 12‑million‑token context window with a 52.2× speedup and only 5% of Opus's cost, as demonstrated on MRCR v2, RULER, and SWE‑Bench benchmarks.
LLMSubQlong context
0 likes · 14 min read

Traditional full‑attention cannot handle million‑token contexts due to exponential compute and memory growth, but DeepSeek V4’s Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) compress, sparsely index, and precisely compute tokens, cutting KV cache to 10% and FLOPs to 27% while enabling a 1‑M token window on a single GPU.
Attention MechanismCSAHCA
0 likes · 12 min read

The article analyzes how both DeepSeek V4 and Meituan's LongCat‑2.0‑P preview, each with trillion‑scale parameters and 1 M‑token context, were trained and inferred entirely on Chinese‑made accelerators, detailing memory optimizations, deterministic operators, MoE redesigns, and massive multi‑card clusters that prove domestic compute can meet top‑tier AI workloads.
Deterministic OpsDomestic AI ChipLongCat
0 likes · 13 min read

DeepSeek V4’s preview launch unveils two open‑source LLM variants—V4‑Pro with 1.6 T parameters and V4‑Flash with 284 B—both supporting a default 1 M‑token context, and introduces novel mHC residual scheduling, hybrid CSA/HCA sparse attention, and Muon optimizer tricks that together deliver top‑tier performance rivaling closed‑source models across coding, long‑text, and reasoning benchmarks.
DeepSeekarchitecturelarge language model
0 likes · 10 min read

DeepSeek has unveiled the V4 preview, offering two open‑source large language models—Pro (1.6 T parameters) and Flash (284 B)—both supporting 1 million‑token context, sparse‑attention efficiency gains, top‑ranked Agent capabilities, and competitive reasoning performance, marking a major milestone for Chinese AI.
1M token contextAgentDeepSeek
0 likes · 5 min read

DeepSeek has launched the open‑source DeepSeek‑V4 series, offering Pro and Flash models with a 1 million token context window, a novel sparse attention mechanism, performance that rivals Opus 4.6 on coding and knowledge benchmarks, tiered pricing, and future cost reductions once Ascend 950 supernodes become widely available.
1M contextAI benchmarkingdeepseek-v4
0 likes · 5 min read

DeepSeek V4 introduces two open‑source MoE models—Pro and Flash—with up to 1.6 T parameters, 1 M token context, a new DSA sparse‑attention mechanism, extensive benchmark results, and a tiered pricing scheme, while remaining compatible with OpenAI and Anthropic APIs.
DeepSeeklarge language modelmillion-token context
0 likes · 9 min read

DeepSeek released the V4 preview, introducing V4‑Pro (1.6 T parameters, 49 B activation neurons, 33 T tokens) and V4‑Flash (284 B parameters, 13 B activation neurons, 32 T tokens) with 1 M token context, a novel DSA sparse attention that reduces compute and memory, and performance that rivals top closed‑source models in agentic coding, world‑knowledge and reasoning benchmarks, while offering an API compatible with OpenAI and Anthropic.
DeepSeekOpenAI API Compatibilitylarge language model
0 likes · 5 min read


ArcThinkApr 11, 2026 · Artificial Intelligence
Developers can sift through official silence and industry leaks—internal statements, Ascend 950PR supply‑chain hints, and sparse‑attention innovations—to assess DeepSeek V4’s likely technical leaps, from million‑token context to native Ascend training, and its strategic impact on the open‑source AI landscape and CUDA independence.
AI model analysisDeepSeekHuawei Ascend
0 likes · 27 min read

This article reviews and compares the most important attention variants used in modern large language models—including multi‑head attention, grouped‑query attention, multi‑head latent attention, sparse and sliding‑window attention, gated attention, and hybrid designs—detailing their motivations, memory trade‑offs, example architectures, and experimental findings.
GQALLMMHA
0 likes · 29 min read


SuanNiFeb 23, 2026 · Artificial Intelligence
GLM‑5, the 744‑billion‑parameter open‑source LLM, introduces DeepSeek Sparse Attention, Multi‑latent Attention, Muon Split optimizer, and a fully asynchronous agentic reinforcement‑learning framework, achieving state‑of‑the‑art performance on long‑context, code, math, and multimodal benchmarks while running efficiently on domestic Chinese chips.
GLM-5asynchronous reinforcement learningbenchmark
0 likes · 12 min read

MiniCPM‑SALA introduces a hybrid sparse‑linear attention architecture that reduces quadratic compute and memory costs, achieves state‑of‑the‑art performance on long‑context benchmarks, and delivers up to 3.5× faster inference than full‑attention models on sequences up to 1 million tokens.
LLMLinear attentionlong context
0 likes · 17 min read

The article presents SALA, a sparse‑linear hybrid attention architecture that replaces full attention in 9B‑parameter models, achieving comparable accuracy while cutting compute and memory costs, enabling million‑token inference on a single RTX 5090 and delivering up to 3.5× speed‑up over Qwen3‑8B.
Hybrid Position EncodingLLM efficiencyLinear attention
0 likes · 18 min read

Z.ai released the open‑source GLM‑5 model with 744 billion parameters, 28.5 T tokens of training data, and new Sparse Attention and Slime RL infrastructure, achieving top open‑source rankings and near‑Claude Opus 4.5 performance on Vending Bench 2 and CC‑Bench‑V2 while adding multi‑scenario agent capabilities.
Agentic engineeringGLM-5benchmark
0 likes · 6 min read

The GLM-5 architecture, uncovered from a GitHub PR, doubles the previous model to 745 B parameters, adopts DeepSeek‑V3 sparse attention and multi‑token prediction, features a 78‑layer MoE with 256 experts, supports a 202K‑token context window, and its rumored test model "Pony Alpha" sparked a 60% rise in Zhipu AI's stock amid a crowded AI release season.
AI Stock ImpactDeepSeekGLM-5
0 likes · 6 min read

This article reviews recent 2025 advances in long‑sequence LLM inference, covering Kimi Linear attention, DuoAttention and DeepSeek Sparse Attention, MegaKernel and MPK designs for kernel‑level efficiency, reinforcement‑learning rollout optimizations, and the Tawa deep‑learning compiler framework.
Deep Learning CompilerLLM OptimizationLinear attention
0 likes · 22 min read

This article explains how a hierarchical sparse‑attention framework redesigns KVCache storage across GPU, CPU, and remote memory, eliminates bandwidth and capacity bottlenecks, and enables efficient inference for 128K‑token and larger contexts with dramatically reduced GPU memory usage and higher throughput.
Dynamic Sparse AttentionGPU memory optimizationHierarchical Storage
0 likes · 20 min read

The article explains how Baidu’s Baige team integrated a Context Parallelism strategy into DeepSeek V3.2, detailing the DSA architecture, the limitations of traditional tensor and sequence parallelism, and how CP distributes computation and memory across GPUs to achieve up to an 80 % reduction in token‑to‑first‑token latency for ultra‑long 128K‑token contexts.
Context ParallelismDeepSeekLLM
0 likes · 9 min read

The article explains how the newly merged Context Parallelism (CP) technique in SGLang, combined with DeepSeek V3.2's Sparse Attention architecture, reduces first‑token latency by up to 80% and alleviates memory pressure for ultra‑long 128K‑token sequences, detailing both algorithmic innovations and engineering solutions.
AI infrastructureContext ParallelismLLM
0 likes · 10 min read

This report analyzes the memory bottleneck of DeepSeek‑V3.2‑Exp’s sparse‑attention decoder, proposes the Expanded Sparse Server (ESS) to offload the latent cache to CPU memory, and demonstrates through high‑fidelity simulation that the approach dramatically improves decode throughput while keeping latency within acceptable limits.
Cache offloadGPU memoryLLM inference
0 likes · 20 min read

DeepSeek's V3.2 release introduces a dual‑model lineup, a Sparse Attention architecture that halves long‑context inference cost, a post‑training reinforcement‑learning pipeline that exceeds 10% of pre‑training compute, and a revamped agent framework that dramatically improves tool‑use and reasoning performance across benchmarks.
Agentic AIDeepSeeklarge language model
0 likes · 11 min read

This article provides a detailed, easy‑to‑understand analysis of DeepSeek‑Math‑V2’s self‑verification training method and DeepSeek‑V3.2’s GRPO framework, sparse‑attention DSA mechanism, massive agent data pipeline, and benchmark results that place both models among the world’s top open‑source large language models.
DeepSeekGRPOLLM
0 likes · 19 min read

This report details the analysis of memory bottlenecks in DeepSeek‑V3.2‑Exp, proposes the Expanded Sparse Server (ESS) that offloads latent cache to CPU memory, and demonstrates through high‑fidelity simulation that the approach, combined with cache‑warmup and overlap techniques, can double decoding throughput for long‑context inference.
Cache offloadGPU‑CPU optimizationLLM inference
0 likes · 21 min read

The Baidu Baige team discovered that DeepSeek‑V3.2‑Exp’s long‑context performance lagged behind the official report, traced the issue to a subtle RoPE layout mismatch in the open‑source inference demo, collaborated with DeepSeek to fix it, and verified that the model’s speed and accuracy fully recovered across multiple benchmarks.
AI infrastructureDeepSeekLLM inference
0 likes · 9 min read

This article reviews the emerging post‑Transformer research landscape, covering linear state‑space models, efficient attention approximations, MLP/conv/RNN hybrids, sparse and causal attention mechanisms, and outlines future trends that may complement or replace the classic Transformer architecture for handling ultra‑long sequences.
AIState Space ModelTransformer
0 likes · 17 min read

InfLLM‑V2 introduces a zero‑parameter, train‑efficient sparse‑attention framework that dramatically speeds up long‑sequence processing while requiring only 5 B tokens for training, and the open‑source MiniCPM4.1 model demonstrates comparable performance to dense attention on both long‑text understanding and deep‑thinking benchmarks.
EfficiencyInfLLM-V2MiniCPM4.1
0 likes · 10 min read

The article explains how DeepSeek’s Lightning Indexer acts as a memory‑filtering expert that computes index scores, selects the top‑k relevant tokens, and maps a compact formula to FP8 kernel code, reducing attention complexity from 128K to 2048 tokens for massive sequences.
DeepSeekFP8Lightning Indexer
0 likes · 7 min read

The article introduces DeepSeek-V3.2, highlighting its new DeepSeek Sparse Attention (DSA) that boosts training and inference efficiency by up to 50%, cuts model usage costs dramatically, explains the updated API endpoints, and details the four‑stage post‑training pipeline that underpins the model’s performance improvements.
AI ArchitectureDSADeepSeek-V3.2
0 likes · 8 min read

DeepSeek‑V3.2‑Exp, the latest experimental large‑language model, is open‑sourced with a paper, featuring a million‑token context window, a new sparse attention mechanism, GRPO‑enhanced reasoning, and detailed cost‑analysis showing up to ten‑fold inference savings.
DeepSeekGRPOInference Optimization
0 likes · 5 min read

A new study from Meta and NYU reveals a two‑step technique—SAMD to locate concept‑specific attention heads and SAMI to scale their influence—enabling precise, low‑cost editing of transformer models for tasks ranging from factual recall to safety control.
AI safetyModel Editingconcept control
0 likes · 11 min read

This article dissects the major bottlenecks of large‑language‑model serving—prefill vs. decode, sparsity, memory bandwidth, KV‑cache growth—and walks through concrete engineering tricks such as paged attention, radix‑tree KV caches, compressed attention, speculative decoding, FlexGen weight scheduling, FastServe queuing, plus a runnable vLLM code snippet.
FastServeFlexGenInference Optimization
0 likes · 18 min read


AIWalkerFeb 25, 2025 · Artificial Intelligence
Sliding Tile Attention (STA) replaces costly full‑3D attention in video DiT models with a block‑wise sliding‑window scheme, achieving up to 10× attention speedup and a 3.53× end‑to‑end generation boost for HunyuanVideo without quality loss, as demonstrated by extensive benchmarks and kernel analyses.
GPU optimizationHunyuanVideoSliding Tile Attention
0 likes · 16 min read


ArchitectFeb 24, 2025 · Artificial Intelligence
The article details the development, architectural evolution, and practical challenges of MoBA—a sparse attention framework inspired by Mixture‑of‑Experts that scales LLM context length to 10 M tokens, supports seamless switching between full and sparse attention, and is now released as a minimal open‑source solution.
AI ArchitectureContext ParallelLLM training
0 likes · 13 min read

The article introduces MoBA, a Mixture‑of‑Block‑Attention mechanism that applies Mixture‑of‑Experts principles to transformer attention, enabling efficient long‑context processing for large language models while maintaining performance comparable to full attention through sparse, trainable block selection and seamless switching.
Attention MechanismLLMMoBA
0 likes · 12 min read

The NSA mechanism introduces a three‑branch hardware‑optimized sparse attention architecture—token compression, token selection, and sliding window—combined with learnable gating to balance global and local context, dramatically improving inference speed and efficiency for long‑context large language models.
AI ArchitectureDeepSeekhardware acceleration
0 likes · 5 min read

This issue reviews China's first open‑source short‑film model SkyReels‑V1, DeepSeek's Native Sparse Attention breakthrough, xAI's massive Grok‑3 deployment on 200k H100 GPUs, and a knowledge‑graph‑guided RAG framework, highlighting their performance gains, architectural innovations, and industry impact.
AIModel EfficiencyRAG
0 likes · 15 min read


AIWalkerFeb 19, 2025 · Artificial Intelligence
DeepSeek introduces the NSA sparse attention mechanism, combining dynamic hierarchical sparsity, coarse token compression and fine token selection to achieve up to 11.6× faster inference, lower pre‑training cost, and superior benchmark performance across general, long‑context, and chain‑of‑thought tasks.
DeepSeekLLM OptimizationNSA
0 likes · 9 min read

The DeepSeek paper on Native Sparse Attention (NSA) presents a hardware‑aligned, trainable sparse‑attention architecture that slashes O(n²) costs, delivers up to 11.6× speedups and 2.3‑point accuracy gains on long‑context benchmarks, and reduces training expense by 47% while scaling to 64k tokens.
DeepSeekGPU optimizationLong-context modeling
0 likes · 11 min read

This article examines DeepSeek's advanced Transformer‑based architecture, dynamic routing, MoE system, multi‑stage training, efficient inference, multimodal capabilities, real‑world applications, technical challenges, and future prospects, providing a comprehensive technical analysis of the model's strengths and limitations.
AI ArchitectureDeepSeekTransformer
0 likes · 15 min read

The SASA model, a structure‑aware sparse‑attention Transformer developed by Alibaba Cloud PAI and Prof. Gao Ming’s team, improves long‑code sequence processing by sparsifying self‑attention using top‑k frequency and AST pattern matrices, achieving higher performance and lower memory/computation costs on CodeXGLUE benchmarks.
ASTCode UnderstandingLong Sequences
0 likes · 8 min read

An in‑depth Q&A breaks down core BERT concepts—from the purpose of the [CLS] token and masking strategies to self‑attention complexity, sparse attention tricks, subword handling of OOV words, warm‑up learning rates, GPT’s unidirectional nature, and ALBERT’s parameter sharing—providing concise explanations for each.
BERTSelf-AttentionSubword Tokenization
0 likes · 7 min read
