Paradigm Shifts in Large Language Models: From Pre‑training to AGI and Future Research Directions

This article examines the rapid development of large language models (LLMs) and identifies two major paradigm shifts: the move from deep learning to two‑stage pre‑training models (BERT/GPT) and the transition toward general artificial intelligence driven by ever‑larger models.

It discusses how scaling laws affect model performance, the importance of training data volume versus model size, and the emergence of “emergent abilities” that appear only after a certain parameter threshold.

The paper analyzes how LLMs store linguistic and factual knowledge within Transformer layers, especially the feed‑forward networks acting as key‑value memories, and explores methods for editing or updating stored knowledge.

Prompting techniques such as zero‑shot, few‑shot, Chain‑of‑Thought (CoT), Self‑Consistency, and Least‑to‑Most are reviewed, showing how they unlock reasoning capabilities without changing model parameters.

It highlights the impact of incorporating code into pre‑training, which dramatically improves reasoning performance, and compares different model families (GPT‑3, Codex, PaLM, Chinchilla) on benchmark tasks.

Future research directions are proposed, including scaling model size further, enhancing complex reasoning, expanding multimodal capabilities, improving human‑LLM interfaces, building high‑difficulty evaluation suites, improving data quality and diversity, and adopting sparse Transformer architectures to reduce training costs.