Baidu's ERNIE-5.5-Flash-Base: 280B MoE Hits Same-Size SOTA with 14B Active Params

Baidu has released and open-sourced ERNIE-5.5-Flash-Base , a sparse mixture-of-experts (MoE) base model with 280B total parameters and 14B activated parameters per token . The model was pre-trained from scratch on 33.5 trillion tokens with a maximum training context of 128K , while fine-tuning and inference support up to 1M tokens .

Benchmark Results

In head-to-head evaluations against same-scale peers — DeepSeek-V4-Flash-Base (284B-A13B) , Hy3-preview-Base (295B-A21B) , and MiMo-V2.5-Base (310B-A15B) — ERNIE-5.5-Flash-Base claims same-size SOTA , with particularly strong performance in reasoning, coding, academic subjects, and long-context tasks.

Model Architecture

The model uses a 43-layer Transformer with hidden size 4096 and 64 attention heads . Its sparse MoE design provides large model capacity while keeping per-token compute controlled.

Three Core Innovations

1. Latent-Gated Sparse Attention (Hybrid HCA + MLA)

To address KV Cache and attention cost growth with long contexts, the model combines two attention types across layers:

37 layers use HCA (Heavy Compression Attention) : each 128 tokens are compressed into a single cache entry; attention computes only over these compressed entries plus a small recent uncompressed window. This gives each HCA layer a global view with a fraction of full-context KV Cache.

6 layers use MLA (Multi-head Latent Attention) with DSA (Dynamic Sparse Attention) : a lightweight indexer scores historical positions for each query; core attention runs only on the top-K positions, avoiding linear cost growth with context length. Baidu reports pre-training loss matches dense attention while training speed improves significantly.

Compared to an HCA+CSA (Compressed Sparse Attention) alternative, allocating the KV Cache budget to a few full-token MLA layers outperforms spreading it over many compressed layers. Despite only 6 MLA-DSA layers, the model achieves solid results on 6 long-context benchmarks.

Two intra-attention refinements further improve efficiency:

Low-Rank Gated Attention : gates are computed from the attention layer's own low-rank query latent (1/4 hidden size), cutting gate projection parameters by 75% while improving model quality in ablation.

Virtual Head Attention (VHA) : adds lightweight cross-head low-rank mixing with identity skip connections after attention, enhancing inter-head information fusion; at inference this folds into the attention operator with negligible overhead.

2. Multi-Router Latent MoE

Standard MoE uses a single router with hard top-K selection. Baidu found router scores correlate only weakly with actual expert utility, especially in middle/deep layers; once an expert is missed it receives no gradient, causing self-reinforcing errors.

Multi-Router Latent MoE employs multiple independently initialized sigmoid routers. An expert can be selected if strongly favored by any router or jointly recognized by several, providing multiple independent judgment perspectives. This raises router-utility correlation (largest gains in middle/deep layers) without increasing expert compute — activated experts, expert FLOPs, and communication volume remain identical to single-router MoE.

Expert computation is also compressed: tokens project from 4096 to 2048-dimensional latent space , experts compute there, then a shared up-projection maps back. This roughly halves expert FLOPs and dispatch/combine communication . The MoE blocks (all FFN layers except the first) use 512 routed experts, top-10 selection, plus 1 shared expert , with auxiliary-loss-free bias correction for load balancing. Stable LatentMoE adds pre-up-projection RMSNorm and SiTU-GLU activations to maintain output scale across 43 layers, and each routed expert's weight is multiplied by its own learnable scalar.

3. ECHO: Backbone Reuse for Future-Token Prediction

Traditional multi-token prediction (MTP) adds dedicated modules whose losses don't directly train the backbone. ECHO reuses the backbone's final layer to predict the next future token, so multi-step prediction loss directly updates the parameters that generate the next token, forcing the backbone to learn sequence continuation earlier.

Baidu reports single-step ECHO pre-training yields clear gains on reasoning and coding. The same layer doubles as a built-in draft model for speculative decoding : trained to predict only one extra token, it can draft multiple consecutive tokens at inference with higher acceptance rate than an equivalently trained external MTP module, linking training-time future prediction to inference-time acceleration across the model lifecycle.

Additional Improvements

MultiMax output head : learnable piecewise temperatures modulate token outputs, suppressing unreasonable long-tail tokens while preserving diversity among plausible candidates; computed in a fused cross-entropy kernel with negligible training overhead and no extra memory, significantly lowering pre-training loss.

R2L tokenizer : new BPE vocabulary of ~200k; numbers are split right-to-left in groups of up to three digits, aiding arithmetic and digit-heavy text.

Training infrastructure : distributed Muon optimizer with FP8 mixed precision; FlashMask-4 attention operator optimized for large head dimensions, improving communication, memory utilization, and compute efficiency at scale.

Data Construction: Three Pillars

Web data : massive multilingual crawl, high-fidelity extraction, model-driven multi-dimensional quality filtering for broader knowledge coverage and higher signal density.

Code data : beyond static repo/web code, includes commits, issues, pull requests, code reviews, requirement-to-PR traces, multi-commit trajectories, auto-generated tests, and CI-verified fixes — strengthening repository-level engineering capability.

Long-document data : natural long documents filtered for long-range dependencies and quality; concatenated semantically similar texts with distractor passages; synthetic MRCR-style retrieval, citation-based QA, and multi-hop QA — building long-context understanding that is hard to recover later with small data additions.

The article emphasizes that repository-level code and long-range dependency abilities need strong pre-training foundations; later small-data supplementation cannot fully compensate.

Open-Source Release & Outlook

The released checkpoint is the pre-training endpoint (no mid-training or post-training), giving developers a unified starting point for domain adaptation, instruction tuning, reasoning RL, and Agent training, as well as controlled comparison of training methods. The model is supported in vLLM and SGLang .

Baidu indicates the ERNIE-5.5-Flash official version (post-trained) will follow soon. The article frames the competition as shifting from raw parameter scaling to per-token activation efficiency, long-context processing cost, and base-model reusability — arguing that embedding efficiency into pre-training grants the initiative in subsequent iteration and deployment.