NextLat: World Model Constrains LLM Reasoning, 3.3x Faster Speculative Decoding
Microsoft Research and MIT propose Next Latent Prediction Transformer (NextLat), which adds a latent state prediction loss during training to compress history into belief states, achieving 98.7% valid route generation on Manhattan taxi data, 54.8% accuracy on Countdown arithmetic, and up to 3.3x speculative decoding speedup on a 1.3B model.
Paper Overview
The paper Next Latent Prediction Transformer (NextLat) appears in NeurIPS 2026. First author Jayden Teoh (MIT PhD) and ten co-authors from Microsoft Research introduce a method that constrains language model reasoning with a world model by predicting the next hidden latent state during training. Corresponding authors are John Langford and Alex Lamb.
Core Method: NextLat
NextLat adds a small network pψ (three-layer MLP with GELU activation) that takes the current hidden state hₜ and the next token embedding, applies layer normalization, and outputs a delta added to hₜ to produce a predicted next hidden state ĥₜ₊₁. Two auxiliary losses are introduced:
Smooth L1 loss between ĥₜ₊₁ and the true hidden state hₜ₊₁ computed by the Transformer at the next position. The true hₜ₊₁ is treated as a constant (stop-gradient); gradients flow back through pψ into the Transformer, shaping its hidden states.
KL divergence between the output distributions obtained by feeding ĥₜ₊₁ and the true hₜ₊₁ through the same output head. The output head is frozen; only ĥ is encouraged to decode the same next-token distribution as the true hidden state.
Total loss = cross-entropy + λ<sub>next-h</sub> × Smooth L1 + λ<sub>KL</sub> × KL. Prediction horizon d > 1 rolls out pψ autoregressively using its own predictions paired with ground-truth tokens, comparing each predicted ĥ to the corresponding true hidden state and averaging.
Theoretical Guarantee
The paper proves that if two conditions hold — (1) hₜ predicts the next token as well as the full history, and (2) hₜ₊₁ is determined by hₜ and the next token as well as the full history — then hₜ is a belief state compressing all predictive information. The proof inducts backward from the sequence end; d=1</sub> suffices, larger <code>d provides denser supervision. Standard next-token training lacks this guarantee because self-attention can always attend back to past tokens, removing pressure to compress history into a single state.
Manhattan Taxi Route Experiments
Dataset: 91 million random taxi trajectories from Vafa et al. (2024), trained for 6 epochs (original work used 1 epoch). Models compared: GPT (next-token), MTP (Multi-Token Prediction), JTP (Joint Multi-Token Prediction), and NextLat (prediction horizon d=8). BST omitted due to cost.
Results (Table 1):
Legal turn test : all models 100%.
Valid trip generation from unseen start/end points : GPT 97.0%, MTP 98.1%, JTP 97.1%, NextLat 98.7% .
Effective rank (lower = more compressed): GPT 160.1, MTP 57.7, JTP 215.8, NextLat 52.7 .
Robustness to 75% random detours : GPT 85.0%, MTP 95.0%, NextLat 95.0% .
Path-Star Graph Navigation
Task: given edge list, start and end nodes, output a path. Difficulty lies only at the central branching node (labeled "Difficult Token"). Three graph families: G₂,₁₀, G₅,₅, G₇,₇ (subscripts = arms × arm length). Training samples fixed at 200k, node values 1–100 (harder than original BST/JTP papers).
On G₇,₇, GPT, BST, MTP, JTP all ~10-20% solve rate; NextLat >90% . On G₂,₁₀ and G₅,₅, BST and NextLat near 100%, while MTP and JTP show high variance across seeds.
Countdown Arithmetic Reasoning
Task: combine given numbers with +−×÷ to reach a target. 500k training problems, 10% targets held out for OOD test.
d=1 : GPT 33.1%, BST 42.3%, MTP 39.2%, JTP 39.0%, NextLat 54.8% .
d=8 : MTP 57.3%, NextLat 58.7% (gap 1.4%).
Ablation (Appendix, Countdown only): Smooth L1 alone 50.7%, KL alone 38.2%, combined 54.8% — Smooth L1 is the primary driver.
Speculative Decoding Speedup
At inference, pψ is used to draft multiple tokens: from hₜ predict Xₜ₊₁ via output head, then pψ computes hₜ₊₁, decode Xₜ₊₂, etc., without calling the large Transformer. Drafts verified using Leviathan et al. (2023) speculative sampling. MTP limited to training horizon d; NextLat can roll out further.
On 1.3B model (FineWeb-Edu validation):
NextLat (d=2) speedup rises with draft length to 3.3× at length 10; d=1 plateaus at ~2.7×.
MTP ~1.7×, JTP ~1.9× (only plotted to length 3).
Cumulative acceptance rate for NextLat (d=2): position 4 ~0.85, position 7 ~0.38, position 10 ~0.09 — showing coherence beyond training horizon.
Domain breakdown (Table 4): Books up to 3.32×, Code 2.38× (best draft length 2–10 per domain).
Language Modeling Metrics
Average accuracy across 8 tasks (1.3B): GPT 58.82, NextLat (d=2) 59.21, (d=1) 58.79 — improvement limited and inconsistent. FineWeb-Edu perplexity: GPT 10.52, NextLat 10.83/10.88, JTP 11.08/11.18 — NextLat better than JTP but slightly worse than GPT.
A₅ Permutation State Tracking
Task: track a permutation of 5 elements through a sequence of even permutations. Training length 12, evaluation up to 36.
NextLat's latent dynamics model (2.62M params) maintains >95% accuracy to length 36, while both the NextLat Transformer (6.43M) and GPT models drop to zero after length 12. The small network learns a computation the Transformer cannot express — the "world's hidden state".
Limitations & Open Questions
Largest scale tested: 1.3B parameters. Whether KL term and d>1 remain necessary at larger scales is unknown. pψ is a simple MLP; more expressive dynamics models unexplored.
Smooth L1 loss rises during pretraining, suggesting sensitivity to optimization.
No comparison with Liu et al. (2024) MTP variants.
Draft length per prompt fixed; adaptive drafting not studied.
Resources
Paper: https://arxiv.org/abs/2511.05963 (v5)
Author blog: https://jaydenteoh.github.io/blog/2026/nextlat Code: https://github.com/JaydenTeoh/NextLat (MIT license)
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
