Tagged articles

SWA

2 articles · Page 1 of 1

Sep 14, 2026 · Artificial Intelligence

DeepSeek V4.1 Flash Architecture: The State Lifecycle Behind 1M Context

This article dissects DeepSeek V4.1 Flash's architecture for 1M-token context, explaining how CED reduces prefill compute, CSA2 shares global KV across layers, hierarchical sparse indexing narrows search, FP4 quantization shrinks storage, SWA uses bounded replay for recovery, and Engram adds a conditional memory path—revealing five cost dimensions of long-context deployment.

Bounded ReplayCEDCSA2

0 likes · 21 min read

DeepSeek V4.1 Flash Architecture: The State Lifecycle Behind 1M Context

NewBeeNLP

Jun 12, 2024 · Artificial Intelligence

Beyond Cosine Decay: Fixed LR + Quick Decay Beats Traditional Schedules in LLM Training

The article analyzes why the traditional cosine decay learning‑rate schedule hinders continued training of large language models and shows that fixed‑learning‑rate strategies such as Warmup‑Stable‑Decay, Cooldown, SWA, and Schedule‑Free Optimizer can match or surpass cosine performance while being more friendly to fine‑tuning.

CooldownLLM-trainingSFO

0 likes · 7 min read

Beyond Cosine Decay: Fixed LR + Quick Decay Beats Traditional Schedules in LLM Training