Can 99% Sparse Transformers Run Faster? Insights from the ‘Attention Is All You Need’ Authors

The paper shows that applying lightweight L1 regularization can make over 99% of FFN activations zero, and by using a new tile‑wise ELLPACK (TwELL) format together with a hybrid routing scheme, inference speed improves up to 30% while memory usage drops over 24% and energy consumption is reduced, all with negligible impact on downstream task performance.

CUDAGPU OptimizationHybrid Routing

0 likes · 8 min read

Can 99% Sparse Transformers Run Faster? Insights from the ‘Attention Is All You Need’ Authors

Machine Learning Algorithms & Natural Language Processing

May 9, 2026 · Artificial Intelligence

Can 99% Sparse Transformers Run Faster? Insights from the Original Authors

A new ICML 2026 paper by Sakana AI and NVIDIA shows that applying lightweight L1 regularization can make Feed‑Forward Network activations in Transformers over 99% sparse, and with the TwELL storage format and a hybrid routing scheme this sparsity translates into up to 20.5% inference speedup, 21.9% training‑step acceleration, lower energy consumption and reduced peak memory across 0.5‑2 B‑parameter models while preserving downstream performance.

CUDAGPU OptimizationHybrid Routing

0 likes · 9 min read

Can 99% Sparse Transformers Run Faster? Insights from the Original Authors

PaperAgent

Feb 16, 2026 · Artificial Intelligence

Why Qwen3.5-Plus Sets a New Standard for Open-Source Multimodal AI

Qwen3.5-Plus, Alibaba’s newly open-sourced multimodal LLM, combines a 397 B parameter model with only 17 B active parameters, leveraging native multimodal training, gated attention, sparse MoE, and FP8 precision to outperform GPT-5.2 and Gemini-3-Pro across vision, reasoning, and agent benchmarks.

Open-sourceSparse Activationgated attention

0 likes · 6 min read

Why Qwen3.5-Plus Sets a New Standard for Open-Source Multimodal AI

Data Thinking Notes

Apr 6, 2025 · Artificial Intelligence

Why Mixture of Experts (MoE) is Revolutionizing Large AI Models

Mixture of Experts (MoE) leverages dynamic conditional computation and specialized expert networks to overcome the parameter explosion and inefficiency of dense models, offering scalable capacity, multi‑task adaptability, and improved efficiency, while addressing challenges such as training stability, communication overhead, and load balancing.

Deep LearningDynamic RoutingMixture-of-Experts

0 likes · 7 min read

Why Mixture of Experts (MoE) is Revolutionizing Large AI Models