Tagged articles
efficient attention
14 articles · Page 1 of 1
Ling‑3.0‑flash demonstrates that a 124‑billion‑parameter model can achieve flagship‑level performance while activating only 5.1 billion parameters, thanks to native mixed‑linear attention, KDA, and extreme MoE sparsity, making it a fast, cost‑effective execution engine for Agent‑centric workflows.
Agent ExecutionLing-3.0-flashMoE
0 likes · 17 min read

The ViT³ paper, a CVPR 2026 best‑paper finalist, introduces test‑time training to compress visual context, achieving 4.6× faster inference and 90 % lower GPU memory on 1248×1248 images, while outlining six design principles and demonstrating its adaptability to classification, detection, segmentation, and generation tasks.
CVPR 2026High-Resolution VisionLinear attention
0 likes · 16 min read

DeepSeek V4 introduces a suite of architectural breakthroughs—including mixed‑expert MoE, manifold‑constrained hyper‑connections, CSA/HCA hybrid attention, and FP4 quantization—that slash inference cost by up to tenfold while delivering million‑token context, competitive benchmarks, dual model variants, and a disruptive pricing strategy.
AI model benchmarkAgentic AIFP4 quantization
0 likes · 41 min read

The Latent‑Condensed Attention (LCA) method dramatically cuts KV‑cache memory by 90%, speeds up pre‑fill by 2.5× and reduces decode latency by 1.8× for 128K‑token contexts, while requiring no extra parameters and preserving model performance across diverse LLMs.
KV cache reductionLCALong Context
0 likes · 10 min read

The article provides an in‑depth technical analysis of DeepSeek‑V4, detailing its novel hybrid attention architecture (CSA and HCA), the manifold‑constrained hyper‑connection (mHC), massive KV‑cache reductions, FLOPs savings across token lengths, and the Muon optimizer with Newton‑Schulz orthogonalization, all backed by concrete benchmark tables and code snippets.
DeepSeekKV cache reductionMuon optimizer
0 likes · 61 min read

This article traces the evolution of sequence models from early RNN/LSTM designs through the breakthrough Transformer, its major branches, dense scaling, efficiency‑focused variants, next‑generation linear‑complexity SSMs, and finally multimodal agent architectures, highlighting each stage's strengths, weaknesses, and typical use cases.
AI ArchitectureLLMModel Evolution
0 likes · 12 min read

This article explains how Attention Residuals (AttnRes) replace traditional residual shortcuts with layer‑wise attention, details the mathematical reformulation, design constraints, static‑Q trick, full and block variants, and presents experimental evidence of significant accuracy gains with modest overhead.
AttentionNLPRMSNorm
0 likes · 11 min read

The article reviews the AAAI 2026 paper "Vision Transformers are Circulant Attention Learners", explaining how modeling self‑attention as a Block‑Circulant matrix enables FFT‑based multiplication that cuts the quadratic complexity of standard attention, achieving up to seven‑fold inference speed‑up while preserving accuracy across ImageNet, COCO and ADE20K benchmarks.
BCCB MatrixCirculant AttentionComputer Vision
0 likes · 15 min read

This article reviews the emerging post‑Transformer research landscape, covering linear state‑space models, efficient attention approximations, MLP/conv/RNN hybrids, sparse and causal attention mechanisms, and outlines future trends that may complement or replace the classic Transformer architecture for handling ultra‑long sequences.
AIState Space ModelTransformer
0 likes · 17 min read

The article analyzes the Tensor Product Attention (TPA) method presented at NeurIPS 2025, explaining how it factorizes Q, K, V tensors to drastically reduce KV cache size and attention complexity, and demonstrates superior convergence, lower perplexity, and faster inference on long‑sequence tasks compared with existing attention variants.
KV CacheLong ContextRoPE
0 likes · 11 min read


AIWalkerJan 17, 2025 · Artificial Intelligence
The CLEAR method linearizes pretrained Diffusion Transformers by restricting attention to a local window, reducing attention FLOPs by 99.5%, accelerating 8K image generation 6.3× while preserving quality, and supporting multi‑GPU patch‑wise inference for high‑resolution text‑to‑image synthesis.
Diffusion TransformersLinear attentionclear
0 likes · 21 min read


NewBeeNLPAug 3, 2024 · Artificial Intelligence
This article reviews recent research on extending large language model context windows to millions of tokens, covering SAMBA's hybrid architecture, Contextual Position Encoding (CoPE), RoPE base length theory, Retrieval Head analysis, and the memory‑efficient Infini‑Attention mechanism.
LLM researchLong Contextefficient attention
0 likes · 10 min read

This talk presents recent year’s research on natural language generation, covering the ProphetNet pre‑trained generation model, external‑knowledge integration for generation, non‑autoregressive pre‑training (BANG), the Poolingformer long‑text architecture, EL‑attention for faster decoding, and a new multi‑task generation benchmark.
Knowledge IntegrationNon-autoregressivePretraining
0 likes · 22 min read

The Twins series, a collaboration between Meituan and the University of Adelaide, introduces conditional positional encoding and spatially separable self‑attention to improve efficiency and performance of vision transformers, achieving state‑of‑the‑art results on ImageNet, ADE20K, COCO and high‑precision map segmentation.
ADE20KCOCOConditional Positional Encoding
0 likes · 20 min read
