Inside Tencent Hunyuan Turbo S: Speed, Cost, and Hybrid Mamba Transformer Explained

Key Features of Turbo S

Turbo S, whose name evokes turbocharging and speed, delivers three main advantages over its predecessor: (1) Speed – first‑token latency is reduced by 44% and throughput is doubled; (2) Cost – the cloud API charges only ¥2 per million output tokens, a several‑fold reduction; (3) Effectiveness – training data, model structure, and MoE efficiency have been refined, and a long‑short chain fusion enables better performance on mathematics and code tasks.

Fast‑Thinking vs. Slow‑Thinking Models

Fast‑thinking models such as Turbo S aim to answer the 90% of user requests that can be solved with intuition, providing instant, concise replies, while the remaining 10% of complex queries are routed to slower, deeper‑thinking models (e.g., Hunyuan T1) for more thorough reasoning.

Hybrid Mamba Transformer Architecture

The core innovation is the Hybrid Mamba Transformer , which combines a linear‑attention Mamba component with traditional full‑attention layers. Standard full attention suffers from three drawbacks: quadratic computational complexity, linear growth of KV‑cache memory, and linearly increasing inference latency. Mamba’s state‑space model offers O(1) per‑token complexity, eliminating KV‑cache pressure and reducing latency.

Hybridization mitigates Mamba’s information‑loss on long sequences by inserting full‑attention layers at selected depths, balancing compression with expressive power. The design explores three variables: the proportion of full‑attention, the layers where it appears, and the method of combining the two mechanisms.

Engineering Optimizations

From an engineering perspective, the Mamba component simplifies sequence‑parallel training because only the previous token’s state needs to be communicated, cutting communication overhead dramatically. During inference, Mamba requires only a tiny “card” of state instead of a full KV‑cache, further lowering memory and compute costs.

Linear Attention vs. MLA

Linear attention (including Mamba) and MLA address KV‑cache bottlenecks differently. MLA reduces KV‑cache usage by about 90%, while the Hybrid Mamba approach can cut it by 60‑70% more, achieving extreme efficiency for long‑context generation.

MoE Research and Scaling Laws

The Hunyuan team has pursued MoE (Mixture‑of‑Experts) since 2022, scaling models to trillions of parameters. Innovations include a Share‑Expert structure that routes tokens through a shared expert plus specialized routing experts, improving gradient stability, and a Compensated Routing mechanism that lowers token‑dropping rates from percent‑level to ten‑thousandths, greatly enhancing training stability.

Extensive scaling‑law experiments revealed that, under fixed compute, training data should be roughly 100× the number of activation parameters, and that increasing data volume continues to improve model capability even beyond two‑ or three‑fold expansions. Fine‑grained expert scaling shows that more granular expert splits raise performance ceilings but increase all‑to‑all communication costs.

Low‑Precision Training Findings

When training MoE models with low‑precision arithmetic, the team discovered a new scaling‑law phenomenon: beyond a certain data‑size threshold, larger datasets actually degrade model performance, contrary to conventional expectations.

Long‑Short Chain Fusion

Turbo S integrates long‑chain knowledge from the T1 model with short‑chain responsiveness, using a two‑stage training pipeline and rejection sampling (based on correctness or length) to blend the strengths of both. This fusion markedly improves results on mathematics, code, and logical reasoning tasks that require deep, multi‑step inference.