A Comprehensive Survey of Large Language Models: Background, Capabilities, Key Technologies, and Future Directions

The paper begins with a historical overview of natural language processing, tracing its roots from the Turing test to modern transformer‑based pre‑trained language models (PLMs) and the emergence of large language models (LLMs) when parameter counts exceed critical thresholds.

It highlights the rapid expansion of LLMs from Google T5 to OpenAI GPT series, noting the societal impact of ChatGPT and the acceleration of research in both academia and industry.

A Chinese research team from Renmin University summarizes recent LLM advances across three dimensions—background, key findings, and mainstream technologies—providing a valuable resource for scholars and engineers.

The survey defines LLMs as models with billions of parameters built on the Transformer architecture, emphasizing that scaling model size, data volume, and compute leads to significant performance gains and novel emergent abilities such as in‑context learning, instruction following, and step‑by‑step reasoning.

Key technologies are organized into scaling, training, capability elicitation, alignment tuning, and tool utilization. Scaling involves increasing parameters, data, and compute; training relies on distributed algorithms (e.g., DeepSpeed, Megatron‑LM) and optimization tricks; capability elicitation uses prompts, instruction tuning, and chain‑of‑thought strategies; alignment tuning incorporates human‑feedback reinforcement learning (RLHF) to mitigate toxic or biased outputs; tool use integrates external calculators, search engines, and plugins to overcome the limitations of pure text generation.

The article lists publicly available LLM resources, including model checkpoints, APIs, datasets, and libraries, and presents tables summarizing models with over 10 B parameters and common data sources.

Pre‑training is described as the foundation for LLM abilities, with emphasis on data collection, cleaning, and pipeline processing, followed by a review of dominant model architectures (encoder‑decoder, decoder‑only, prefix‑tuned) and Transformer component configurations.

Model training challenges such as memory constraints and parallelism are addressed through 3D parallelism, ZeRO, and mixed‑precision techniques.

Adaptive fine‑tuning is covered in detail, focusing on instruction tuning (supervised fine‑tuning on formatted instruction data) and alignment tuning (human‑feedback RL to align models with human values).

Usage techniques include prompt engineering, in‑context learning, and chain‑of‑thought prompting, with explanations of when and why these methods improve performance on complex reasoning tasks.

Evaluation is divided into basic tasks (language generation and understanding benchmarks) and advanced tasks (alignment, tool interaction, and external environment integration), illustrated with figures.

The concluding section discusses open challenges and future directions: theoretical understanding of how information is organized in massive networks, architectural innovations for efficiency and longer context windows, more systematic and economical pre‑training methods, improved prompting automation, safety and alignment concerns (hallucinations, bias, misuse), and the broader impact of LLMs on search, assistants, and the pursuit of artificial general intelligence.