Baidu Open-Sources ERNIE 5.5 Flash Base: Efficient 280B MoE for Agents

Model Overview

On November 5, Baidu announced ERNIE 5.5 Flash Base (文心 5.5 Flash Base), a sparse mixture-of-experts (MoE) language model with 280B total parameters and 14B activated per token. The model was pre-trained from scratch on 33.5 trillion tokens, with a maximum training context of 128K tokens, extended to 1M tokens during fine-tuning and inference. The base model weights are openly available in FP8 and BF16 formats on Hugging Face, with integration cookbooks for vLLM and SGLang.

Hugging Face FP8: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base

Hugging Face BF16: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base-BF16

SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/Baidu/ERNIE-5.5-Flash

vLLM cookbook: https://recipes.vllm.ai/baidu/ERNIE-5.5-Flash

The model comprises 43 Transformer layers, hidden size 4096, 64 attention heads. Except for the first layer, all feed-forward networks are replaced by MoE blocks. Pre-training uses the Muon optimizer with FP8 mixed precision; context length is gradually scaled to 128K during training, then externally extended to 1M for fine-tuning and inference.

Benchmark Results

Official comparisons against same-tier base models — DeepSeek-V4-Flash Base (284B/13B), Hy3-preview-Base, and MiMo-V2.5-Base — show ERNIE 5.5 Flash Base leading overall across knowledge, reasoning, math, code, multilingual, and long-context benchmarks. Detailed comparison with DeepSeek-V4-Flash-Base reveals:

HumanEval+: 82.3 vs 69.5

LiveCodeBench: 40.1 vs 31.9

FullStackBench: 54.6 vs 43.3

Code capability shows the largest margin. A few evaluations (e.g., ChineseSimpleQA) show a slight deficit. Long-context benchmarks (six tasks) confirm the architecture retains strong long-text ability despite heavy compression in most layers.

Core Architectural Innovations

1. Hybrid Sparse Attention: MLA + HCA (1/7 Full-Context Access)

Standard Transformers store KV cache for every token at every layer, causing linear memory growth with context length. Baidu's insight: not every layer needs token-level access to the full history.

ERNIE 5.5 Flash Base adopts a mixed scheme:

MLA (Multi-head Latent Attention) layers — 6 out of 43 — retain token-level access to the entire context.

HCA (Heavy Compression Attention) layers — the remaining 37 — compress every 128 tokens into a single cache entry; attention computes only over compressed entries plus a small recent uncompressed window.

During long-context inference, MLA layers perform fine-grained retrieval, while HCA layers maintain a coarse-grained view. Because HCA layers require only a fraction of the KV cache, total cache footprint drops dramatically.

To prevent the six MLA layers from becoming a compute bottleneck, Baidu introduces DSA (Dynamic Sparse Attention) within MLA: a lightweight indexer scores historical positions per query, restricting dense attention to top-k positions. This makes per-query cost independent of context length while matching dense-attention pre-training loss.

Under equal KV-cache budget, the HCA+MLA combination outperforms the more common CSA+HCA (Compressed Sparse Attention + HCA) in pre-training, supporting fine-grained retrieval.

Two additional attention-layer improvements:

Low-rank Gated Attention: Reuses the layer's own low-rank query latent to compute the output gate. Latent dimension is 1/4 of hidden size, cutting gate projection parameters by 75% while improving ablation results.

Virtual Head Attention (VHA): Lightweight cross-head low-rank mixing with identity skip connection, fuseable into kernels at near-zero overhead. The VHA paper Virtual Head Attention was accepted at NeurIPS 2026.

2. Multi-Router MoE: Same Cost, Better Expert Selection

Traditional MoE uses a single router per token, making a hard top-k choice. Router scores correlate weakly with actual expert utility, especially in middle/deep layers; once an expert is missed, it receives no gradient, reinforcing errors.

Multi-Router deploys multiple independently initialized sigmoid routers simultaneously. An expert is selected if any router strongly favors it or several routers moderately favor it. This turns "one router decides" into "ensemble verdict", reducing single-misjudgment impact. Activated expert count, FLOPs, and communication volume remain identical to single-router MoE — no extra inference cost. Official reports show significantly improved router-utility correlation, with the largest gains in previously weakest middle/deep layers.

Configuration details:

512 routed experts per MoE block, top-10 selected, plus 1 shared expert.

Auxiliary-loss-free bias correction for load balancing.

Token projected from 4096 to 2048 hidden dimension; experts compute in this latent space, halving communication and expert FLOPs.

Stable LatentMoE design (RMSNorm + SiTU-GLU) keeps expert output scale stable across 43 layers; each expert's routing weight multiplied by a learnable scalar.

3. ECHO: Future-Token Prediction Without Extra MTP Modules

Multi-token prediction (MTP), popularized by Meta (2024) and adopted in DeepSeek V3, attaches extra Transformer layers to predict multiple future tokens, providing denser supervision and enabling speculative decoding. However, the main trunk still only optimizes next-token prediction; long-range planning is not directly trained.

ECHO eliminates the separate MTP modules. Instead, the final layer of the main trunk simultaneously predicts the next token and one additional future token. Multi-step loss thus backpropagates directly into the parameters that generate the current token, forcing the trunk to plan ahead.

Benefits:

Pre-training with single-step ECHO yields clear gains on math, reasoning, and code tasks.

The same final layer serves as a built-in draft model for speculative decoding, adding virtually no parameters.

Emergent extrapolation: although trained to predict only one extra token, the layer can draft multiple consecutive tokens at inference, achieving higher acceptance rates than equivalently trained external MTP modules.

Additional Component-Level Improvements

Manifold-constrained Hyper-Connections (mHC): Replaces standard residual connections. Each layer reads/writes four parallel residual streams via a constrained mixing matrix, widening inter-layer residual bandwidth at near-zero cost, following the recent hyper-connection trend.

MultiMax Output Head: Addresses the single-temperature softmax trade-off between sparsity and multi-modality by introducing learnable piecewise temperatures. Suppresses unreasonable long-tail tokens while preserving diversity among plausible candidates; negligible training overhead and no extra memory.

New BPE Tokenizer (~200K vocab): Numbers split right-to-left in groups of three digits (e.g., 1,234,567 → 1, 234, 567), aligning digit positions for better arithmetic and number-heavy text handling.

Data Construction

Three focused directions:

Web data: Multilingual crawling, high-fidelity extraction, model-driven quality filtering.

Code data: Beyond static code, includes commits, issues, pull requests, code reviews, automated tests, and CI-verified fixes — targeting repository-level reasoning and agent software engineering, aligned with the Base positioning.

Long-sequence data: Natural long documents, concatenated distractor texts, and synthetic retrieval/multi-hop QA data.

Infrastructure & Operators

Distributed Muon Optimizer: Per-tensor orthogonalization changes communication patterns. Dense parameters routed through same-GPU-index paths across nodes, fused with MoE communication into a single collective operation. Optimizer states sharded with inter-node balance to avoid memory hotspots. Scales efficiently to thousands of GPUs.

FlashMask-4 Operator: Designed for large head dimensions, implemented via CuTe-DSL supporting SM90 (Hopper) and SM100 (Blackwell). Supports qk head dim 576, v head dim 512. On SM100, forward pass up to 175% faster, backward up to 74% faster. Open-sourced in PaddlePaddle's GitHub repository.

Significance & Outlook

ERNIE 5.5 Flash Base validates multiple architectural innovations in a single large-scale pre-training run. The open base checkpoint lets developers direct subsequent domain adaptation, instruction tuning, RL, or agent-behavior shaping — avoiding from-scratch training for local languages, cultures, and industrial scenarios.

The design centers on efficiency for the emerging "Super Agent" era: long-context pre-fill and KV-cache pressure are the primary bottlenecks for high-frequency, long-running agents. Hybrid sparse attention, multi-router MoE, and ECHO directly target these constraints.

Baidu indicates the full ERNIE 5.5 Flash (chat/instruct version) will follow soon. Open weights are the entry ticket; real-world deployment will test whether the base model's potential translates into sustained competitive advantage in an increasingly crowded Flash-tier landscape.