What Will Large Language Models Look Like in the Next Five Years? A Deep Dive into Trends and Challenges

TL;DR

A colleague asked for a three‑to‑five‑year forecast of model demand. Short‑term model changes occur every few months, but long‑term evolution can be inferred by reviewing the past five years, assessing the current landscape, and projecting future trends.

1. Review of Five Years Ago

In 2020 the author explored early quantitative‑trading attempts with RNN/LSTM, noting limited data and compute that constrained model size. The introduction of the Transformer enabled large‑scale parallel sequence processing, which sparked research into distributed object detection, network‑security algorithms (e.g., zadns ), and large‑scale distributed training.

"Algebraic topology and algebraic geometry could simplify computation and improve generalization, but mastering them requires years of study; meanwhile the AI hype cycle often inflates expectations beyond what current models can deliver."

Subsequent work covered the shift to sparse attention (e.g., NSA, MoBA), mixture‑of‑experts (MoE) and mixture‑of‑domains (MoD) architectures, distributed reinforcement‑learning frameworks such as Nimble, intent‑network linguistic models, and reinforcement‑learning‑driven network optimization.

2. Current Status

Scaling dominates the last five years. The Transformer architecture has persisted for over seven years, while sparse‑attention methods (NSA, MoBA) and MoE/MoD variants address scaling limits. Reinforcement learning progressed from GPT‑3 to ChatGPT, with RLHF, reasoning models, and PPO/DPO/GRPO variants.

Key challenges include benchmark over‑optimization (e.g., Llama‑4 failures), hallucinations in reasoning models, and misuse of benchmarks highlighted by the paper “The Leaderboard Illusion”. Successful examples are AlphaFold, Anthropic’s interpretability research, and DeepSeek’s Prover‑V2.

Application ecosystems (MCP, Agent‑2‑Agent) are expanding, but edge‑side large models lag due to limited mobile AI hardware (AIPC). Model instruction‑following and task‑planning capabilities still need improvement.

3. Future Five‑Year Outlook

3.1 Near‑Term (2025‑2026)

Reinforcement learning will remain a hot topic, but excessive score‑chasing can degrade other abilities. DeepSeek’s reward models continue to improve, and upcoming GRM architectures may open new pathways. Distributed inference efficiency will be a primary infrastructure focus.

Agent‑centric ecosystems (MCP) are emerging, raising questions about expressive power and the need for more efficient inter‑model communication.

3.2 Mid‑Term (2026‑2027)

Open‑source models are expected to reach 1‑3 trillion parameters as compute and memory capacity grow.

Sparse‑attention and dynamic memory contexts (e.g., Titan, DeepSeek NSA, Kimi MoBA) will evolve, requiring new KV‑Cache handling and possibly two‑level MoE gating.

Mixture‑of‑Agents (MoA) systems built on distributed reinforcement learning will become common, and interpretability research will guide architecture design.

MLP and MoE blocks will become increasingly sparse. The ratio of routed to activated experts is likely to stabilize around values observed in Qwen‑3 (8/128) and DPSK (8/256). Scaling to 1‑3 T‑parameter models will push activation sizes to 32‑100 B parameters, with layer counts near 60 to balance latency and bandwidth.

3.3 Long‑Term (2028‑2030)

Research will focus on higher‑order mathematical foundations (topology, algebraic geometry, information geometry) to improve model generalization and handle non‑linear stochastic dynamics. Feature decomposition and attribution graphs may inform new categorical MoE routing strategies.

Distributed reinforcement learning may replace a single monolithic “master” model with many 32‑B dense models that self‑organize via reinforcement learning, forming a hierarchical Mixture‑of‑Agents.

4. Infrastructure

Commercial incentives fragment the ecosystem; collaborative algorithm‑infra co‑design is essential. Bandwidth scaling (e.g., NVIDIA ScaleUP) must be balanced against reliability and thermal constraints, as illustrated by Google’s 1 MW cabinet designs. Efficient interconnects (PCIe Gen6, NVL72/NVL144/CM384) and scalable KV‑Cache mechanisms will be critical for distributed inference.

5. Other Considerations

Under‑discussed areas include autonomous driving, embodied intelligence, and AI‑for‑Science. Trends such as NVIDIA reducing FP64 performance for large‑model workloads may impact scientific computing.

References

zadns: https://github.com/zartbot/zadns

The Leaderboard Illusion: https://arxiv.org/html/2504.20879