How Step 3.5 Flash Bridges the Gap to Top LLMs with Sparse Expert Architecture

Step 3.5 Flash, recently released by the Step Star team, is a 196 billion‑parameter large language model with 11 billion activation parameters. Its performance on complex agent tasks rivals frontier models such as Gemini 3.0 Pro, Claude Opus 4.5, and GPT‑5.2 xHigh.

Model Architecture Balances Compute and Intelligence

The model adopts a sparse‑mixture‑of‑experts design focused on sharp reasoning and reliable execution. It interleaves sliding‑window attention (SWA) with full attention in a 3:1 ratio, allowing the model to concentrate computation on the current context window rather than the entire sequence.

To further boost generation speed, a Multi‑Token Prediction (MTP) technique predicts the next three tokens while emitting the current one, effectively reducing latency.

In the sliding‑window layers, the number of query heads is increased from 64 to 96 and combined with per‑head gated attention, which filters out irrelevant information within the window. An expert‑parallel load‑balancing strategy ensures uniform workload across GPU groups, preventing both idle and overloaded devices.

Table 1 (illustrated in the accompanying image) shows that the hybrid attention layout matches or exceeds the baseline full‑attention model with only a modest cost increase. Table 2 confirms the superiority of the gated‑attention mechanism across benchmark tests.

Infrastructure Guarantees Training Stability

The model runs on a super‑computing cluster of 4,096 NVIDIA H800 GPUs. The team built a lightweight, high‑performance training framework called Steptron, which decouples parallelism so that attention and expert modules can use different partitioning strategies.

During training, thousands of GPUs stay synchronized via an optimized data‑transfer pipeline that alternates intra‑node high‑speed links with inter‑datacenter network paths, saving significant time.

A high‑throughput monitoring server records roughly 6 million status metrics per training step, enabling detection of rare “expert death” or activation spikes. When such spikes occur, activation clipping caps values within a safe range.

The training run processed 17.2 trillion tokens smoothly, with only a single isolated loss fluctuation (see Figure 3).

Reinforcement Learning Unlocks Agent Potential

After supervised fine‑tuning (SFT), the model enters a reinforcement‑learning (RL) stage. To handle complex logical reasoning, the team introduced a MIS‑PO filtering strategy that discards low‑quality trajectories and focuses optimization on high‑quality samples.

Two RL subsystems operate independently: RLVR handles tasks with verifiable rewards such as mathematics and code, while RLHF uses a reward model to evaluate textual quality.

Parallel Coordinated Inference (PaCoRe) allows the model to explore multiple reasoning paths simultaneously with minimal latency, then aggregate the best answer.

Training Data and Evaluation

The pre‑training phase starts with broad open‑domain data, then gradually shifts toward code and software‑related corpora. Token window length expands from 4 k to 32 k, and a mid‑stage focuses on 128 k‑token ultra‑long context.

A curated dataset of 871 k samples (72.3 billion tokens) was built from open‑source and real user interactions, with domain distribution detailed in Table 3.

Benchmark results (Tables 4 and 5) show that despite using only 11 billion activation parameters, Step 3.5 Flash matches or surpasses much larger competitors on tasks such as AIME 2025, IMO‑AnswerBench, and SWE‑Bench Verified.

Remaining Challenges and Future Work

While the model excels in generation efficiency and specialized domains, further improvements are needed for extreme long‑dialogue stability and ultra‑specialized tasks. The team plans algorithmic pruning of reasoning paths to enhance structural stability under high‑complexity or ultra‑long conversations.

Overall, the combination of a finely tuned sparse‑expert architecture, innovative reinforcement‑learning strategies, and a robust training infrastructure makes high‑performance LLM capabilities achievable with relatively low compute cost.