Tagged articles
Multi-Token Prediction
19 articles · Page 1 of 1
Soft-MTP decomposes the cross-entropy gradient in multi-token prediction into a distribution-alignment term and a residual term, then cancels the residual from gradients flowing to MTP heads and the shared backbone while preserving cross-entropy training on the shared output projection, improving language modeling and speculative decoding without extra parameters or full KL computation.
Cross-Entropy LossGradient DecompositionKL Divergence
0 likes · 17 min read

The article examines why traditional prompt engineering is no longer sufficient in production AI systems, detailing how harness layers, context reassembly, tool orchestration, token generation methods, training objectives, and architecture choices transform a simple prompt into a complex, multi‑stage workflow that demands robust, system‑level design.
Context ManagementHarnessLLM
0 likes · 16 min read

The PIPO (Pair‑In, Pair‑Out) architecture compresses two input tokens into one latent vector and expands each hidden state to emit an extra draft token, halving effective input length and doubling per‑step output, achieving up to +7.15 pass@4 improvement and 2.64× TTFT / 2.07× TPOT speedups on benchmarks such as AIME 2025, GPQA‑Diamond, LiveCodeBench v6 and LongBench v2 with Qwen3.5‑4B/9B backbones.
EfficiencyMulti-Token PredictionOn-Policy Distillation
0 likes · 14 min read

In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.
4-bit quantizationKV compressionMulti-Token Prediction
0 likes · 13 min read

The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.
LLM PretrainingMCE lossMulti-Token Prediction
0 likes · 9 min read

Token Superposition Training (TST) speeds up large‑language‑model pre‑training by up to 2.5× without altering model architecture or compute budget, using a superposition phase that averages token embeddings into bags and predicts groups of tokens, followed by a standard recovery phase, as demonstrated on 10B‑parameter MoE and smaller models.
LLM PretrainingMCE lossMoE
0 likes · 10 min read

Nous Research introduces Token Superposition Training, which groups tokens into bags, averages their embeddings, and predicts token groups without altering model architecture or adding compute, achieving up to 2.5× faster pre‑training while maintaining standard inference.
LLM PretrainingMCE lossMoE
0 likes · 10 min read

The GLM-5 architecture, uncovered from a GitHub PR, doubles the previous model to 745 B parameters, adopts DeepSeek‑V3 sparse attention and multi‑token prediction, features a 78‑layer MoE with 256 experts, supports a 202K‑token context window, and its rumored test model "Pony Alpha" sparked a 60% rise in Zhipu AI's stock amid a crowded AI release season.
AI Stock ImpactDeepSeekGLM-5
0 likes · 6 min read

Step‑3.5‑Flash, a 196‑billion‑parameter open‑source LLM that activates only 11 B per token via a Mixture‑of‑Experts design, delivers 3‑plus‑times faster inference, matches top‑tier closed‑source models on SWE‑bench and other benchmarks, supports 256 K context, runs on consumer‑grade hardware, and is already integrated into vLLM, SGLang, and Claude Code, though it has known token‑efficiency and domain‑stability limitations.
LLM benchmarkMoEMulti-Token Prediction
0 likes · 11 min read

This article explains DeepSeek's three core innovations—DeepSeekMoE with fine-grained experts and shared experts, Multi-Head Latent Attention (MLA) for compressed KV-cache, and Multi-Token Prediction (MTP) for faster training convergence—detailing how each reduces compute costs and memory overhead while maintaining model performance.
DeepSeekKV cache optimizationLLM Architecture
0 likes · 5 min read

Alibaba's Qwen team unveils Qwen3-Next, a hybrid expert LLM with 800 B parameters but only 30 B active, delivering training costs under one‑tenth of comparable dense models and more than ten‑fold inference throughput for long contexts, while matching or surpassing larger models on benchmark tasks.
AILLMMulti-Token Prediction
0 likes · 9 min read

A recent Hugging Face pull request reveals Alibaba’s upcoming Qwen3‑Next series, highlighting its extreme‑context, parameter‑efficient design that combines a 1:50 high‑sparsity MoE, a hybrid attention architecture mixing gated attention with Gated DeltaNet, and a Multi‑Token Prediction technique, promising ten‑fold throughput gains for 32K‑plus token contexts.
AI ArchitectureMulti-Token PredictionQwen3-Next
0 likes · 8 min read

This article provides a detailed technical analysis of DeepSeek‑V3, covering its MOE architecture, the novel Multi‑head Latent Attention (MLA) mechanism, the DualPipe pipeline‑parallel algorithm, mixed‑precision FP8 training, and the Multi‑Token Prediction (MTP) inference improvements that together boost performance and efficiency.
DeepSeekDualPipeFP8
0 likes · 44 min read

The DeepSeek V3-0324 model, released on March 24, 2025 with 6.85 trillion parameters and a Mixture‑of‑Experts architecture, is fully open‑source on Hugging Face and brings notable upgrades in coding ability, structured responses, stability, generation length, and speed, while offering performance comparable to leading closed‑source models such as Claude 3.7.
AI code generationDeepSeekMulti-Token Prediction
0 likes · 10 min read


AntTechFeb 27, 2025 · Artificial Intelligence
Researchers from Ant Group and Zhejiang University propose K-ON, a multi-token parallel prediction method that enables large language models to perceive knowledge graph entities through entity-level contrastive learning, achieving superior performance, lower cost, and higher efficiency on KG completion benchmarks.
K-ONMulti-Token Predictionentity contrastive learning
0 likes · 8 min read

The article provides an in‑depth overview of DeepSeek's transformer‑based foundation, Mixture‑of‑Experts architecture, novel attention mechanisms, multi‑token prediction, FP8 mixed‑precision training, knowledge distillation, reinforcement‑learning approaches, and compares its performance and cost advantages against leading models such as GPT and Gemini.
AI model architectureDeepSeekFP8 training
0 likes · 29 min read

This article dissects the Multi‑Token Prediction (MTP) technique used in DeepSeek‑R1, contrasting it with traditional next‑token prediction, detailing Meta’s MTP design, DeepSeek’s adapted architecture, loss weighting, and why MTP is applied only during training to boost efficiency and model capability.
DeepSeekMTPMulti-Token Prediction
0 likes · 9 min read

The article enumerates DeepSeek’s extensive technical optimizations—including Grouped Query Attention, Multi‑head Latent Attention, Mixture‑of‑Experts, 4D parallelism, quantization, and multi‑token prediction—that together enable cheap, high‑performance large language models.
4D parallelismDeepSeekGrouped-Query Attention
0 likes · 8 min read

This article reviews the evolution of Multi‑Token Prediction (MTP) techniques—from early blockwise parallel decoding to Meta's and DeepSeek's implementations—explaining their architectures, training and inference workflows, and the speed‑up gains they offer for large language models.
DeepSeekLLMMTP
0 likes · 20 min read
