China Telecom Trains 285B MoE Model on 8192 Ascend 910C Chips, Tops Hugging Face
China Telecom's Xingchen AGI Lab, with Huawei and HIT Shenzhen, pre-trained a 285B-parameter MoE model using 8192 Ascend 910C chips, detailing architecture optimizations (MLA, DSA, IndexShare, DSpark), system-level improvements (HyperParallel-FSDP, Owner-based Muon, selective recomputation), and the TeleGym RL framework for agent tasks, with a trillion-parameter model underway.
China Telecom's Xingchen AGI Lab, in collaboration with Huawei and Harbin Institute of Technology (Shenzhen), has published a technical report describing the pre-training of a 285-billion-parameter Mixture-of-Experts (MoE) model on a cluster of 8,192 Huawei Ascend 910C chips. The report topped the Hugging Face paper list (arXiv:2611.22115v1).
Model Architecture Optimized for Ascend 910C
The primary model is a Decoder-only MoE with 285B total parameters and ~17B activated parameters per token. It has 45 layers: the first 3 use dense FFNs, the remaining 42 use MoE FFNs. Hidden dimension is 4,096; there are 256 routed experts, with top-8 selected per token plus one shared expert. Maximum context length is 1 million tokens.
Attention Mechanism: MLA + DSA + IndexShare
To reduce memory access and redundant computation in long-context scenarios, the team combined Multi-head Latent Attention (MLA), DeepSeek Sparse Attention (DSA), and cross-layer index sharing (IndexShare). MLA compresses KV representations to lower KV cache storage. DSA introduces a Lightning Indexer that selects top-K historical KV states per query, so the main attention only processes a sparse subset. IndexShare reuses indices across layers, avoiding repeated index computation.
On Ascend 910C simulations, the MLA+DSA+IndexShare combination achieved 4.68× prefill throughput and 1.27× decode throughput over an MLA-only baseline.
Depth vs. Width Trade-off
Because Ascend 910C has strong matrix compute but relatively limited vector compute and memory bandwidth, overly deep networks increase non-matrix overhead (normalization, expert routing, indexing, inter-layer scheduling). The team compared a 60-layer narrow-expert network (expert intermediate dim 1,536, 6 active experts) against a 45-layer wide-expert network (intermediate dim 2,048, 8 active experts) under comparable parameter budgets. The 45-layer wide design delivered 20% higher prefill throughput, 11% higher decode throughput, and ~10% higher overall training throughput, so it was chosen.
Speculative Decoding with DSpark
For inference acceleration, the team introduced DSpark speculative decoding. A 5-layer DFlash backbone computes representations for 7 speculative positions in parallel, predicting multiple candidate tokens at once. A lightweight Markov Head performs logit correction to capture inter-token dependencies, and an acceptance confidence head estimates the probability each candidate will be accepted by the main model. The main model then verifies the candidate block and accepts the longest valid prefix, reducing serial token-by-token generation overhead.
On six math reasoning and code generation benchmarks, DSpark improved decode throughput 2.67–3.77× over the base model and 1.48–2.10× over an MTP3 baseline.
Large-Scale Pre-Training System Optimizations
Built on MindSpore's HyperParallel framework, the system targets communication, compute, and memory management on Ascend clusters.
HyperParallel-FSDP with Hybrid Sharded Data Parallelism (HSDP)
Standard FSDP shards parameters across all nodes, requiring AllGather before each layer's forward pass and ReduceScatter after backward, causing frequent cross-node communication. At thousands of chips, cross-SuperPod bandwidth and latency become bottlenecks. HyperParallel-FSDP instead shards parameters via FSDP within each SuperPod and replicates parameters across SuperPods . Frequent AllGather/ReduceScatter operations stay inside the high-speed intra-SuperPod interconnect; only coarse-grained gradient AllReduce crosses SuperPod boundaries. This design leverages data parallelism to relieve per-GPU memory pressure while minimizing cross-node traffic, enabling large-model training without pipeline parallelism (avoiding pipeline bubbles).
Owner-based Muon Optimizer
In sharded training, multiple nodes would otherwise reconstruct full matrices and redundantly run Newton-Schulz orthogonalization. The Owner-based Muon approach assigns each matrix's full orthogonalization to a designated owner node within the shard group; the owner computes the update and broadcasts the result back. This eliminates redundant computation while preserving existing parameter and optimizer state sharding.
Fine-Grained Selective Recomputation
Instead of deciding recomputation at the whole-layer level, this mechanism selects individual operators to retain based on their compute time and output memory footprint. High-cost operator outputs are kept; others are recomputed during backward pass. Activation data management between recomputed and retained regions is further optimized to reduce unnecessary memory residency.
DSA-Specific Optimizations for Long Sequences
Layer-wise Indexer Backpropagation: Processes indexer auxiliary gradients early, shortening the lifetime of related computation graphs.
Q-Block Pairing and Layout Planning: Improves compute and data layout across different query blocks.
Document-Aware KV Window Compression: Reduces invalid KV accesses and memory usage when training on packed documents.
Cluster-Level CPU Affinity and Core Isolation
Training threads are bound to dedicated CPU cores; system services and interrupt handling are moved off the training CPU pool, reducing host scheduling contention that can delay device task submission and collective communication.
With these optimizations, the team successfully completed pre-training of the 285B MoE model on 8,192 Ascend 910C chips.
TeleGym: Unified Reinforcement Learning Platform for Agent Tasks
For post-training, the team built TeleGym, a unified agent execution platform connecting task execution, result evaluation, trajectory construction, and RL optimization. It supports diverse agent harnesses, execution environments, and persistent sessions, handling math reasoning, code generation, and multi-turn agent tasks uniformly.
To address variable task durations and long-horizon interruptions, TeleGym employs asynchronous execution, bounded oversampling, and partial rollout. These mechanisms allow model training and task execution to proceed in parallel; even when the policy updates mid-training, unfinished agent tasks continue and retain necessary trajectory and reward information, improving continuity and resource utilization for complex task training.
Trillion-Parameter Model in Progress
The report reveals that the team is now training a trillion-parameter model in the Xing 4 series on Ascend chips, further exploring training efficiency and system scalability at larger model sizes.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
