Evolution of Language Models: From Statistical N‑grams to GPT‑4

1 Natural Language Understanding and Language Models

1.1 Natural Language Processing

Natural Language Processing (NLP) is a core field of artificial intelligence that can be divided into Natural Language Understanding (NLU) and Natural Language Generation (NLG). The development of NLP can be roughly split into three stages: rule‑based systems before the 1980s, the rise of machine‑learning and neural‑network methods after the 1980s, and the Transformer‑based large‑language‑model era since 2017.

NLU aims to give machines human‑like comprehension abilities, while NLG converts non‑linguistic data into readable text. NLP is often called the "crown jewel" of AI.

1.2 Language Models

A language model defines a probability distribution over sequences of words. Traditional n‑gram models estimate probabilities by counting frequencies, but suffer from data sparsity as n grows. To alleviate this, Bengio (2003) introduced the Neural Network Language Model (NNLM), which laid the groundwork for word embeddings.

Chain‑rule definition:

Parameter space: vocabulary V, sequence length T.

Model parameters grow exponentially with n.

Data sparsity: rare co‑occurrences and out‑of‑vocabulary words.

Evaluation: Practical performance (task‑specific) and perplexity (theoretical).

2 Evolution of Language Models

The paradigm shift moved from rule‑based to statistical, then to neural‑network‑based deep learning, mirroring the overall history of NLP.

2.1 Statistical Language Models

By applying the Markov assumption, n‑gram models reduce the parameter space. Common variants are Unigram (n=1), Bigram (n=2), and Trigram (n=3). Smoothing techniques are used to mitigate sparsity.

When n is large: richer context but exponential parameter growth and high computational cost. When n is small: less context, cheaper computation, but poorer accuracy.

2.2 Neural Network Language Models

The first NNLM was proposed by Bengio (2003). It maps words to low‑dimensional embeddings, alleviating the curse of dimensionality and enabling similarity and analogy reasoning.

Limitations include fixed‑length history and inability to capture long‑term dependencies.

2.3 Recurrent Neural Networks (RNN)

RNNs process sequences recursively, discarding the Markov assumption and allowing each hidden state to depend on the entire previous context.

Drawbacks: gradient vanishing/exploding.

2.4 Long Short‑Term Memory (LSTM)

LSTM introduces gated cells (forget, input, output) to preserve long‑range dependencies and mitigate gradient issues.

2.5 ELMo

ELMo generates context‑dependent word embeddings using a two‑layer bidirectional LSTM, addressing the static nature of earlier embeddings.

2.6 Transformer

Proposed by Google in 2017, the Transformer relies solely on self‑attention, enabling parallel computation and long‑range dependency modeling.

2.6.1 Attention Mechanism

Attention computes similarity between queries (Q) and keys (K), applies softmax, and weights values (V) to produce the attention vector.

2.6.2 Residual Networks (ResNet)

ResNet adds shortcut connections to ease optimization of very deep networks.

2.6.3 Position Embedding

Sinusoidal absolute position encoding injects token order information without learning additional parameters.

2.6.4 Transformer Architecture

Stacked encoder and decoder layers, each containing multi‑head self‑attention and feed‑forward sub‑layers, with residual connections and layer normalization.

3 GPT Series Overview

The GPT family, starting with GPT‑1 (2018) and progressing through GPT‑2, GPT‑3, ChatGPT, and GPT‑4, demonstrates the power of large‑scale pre‑training followed by fine‑tuning or prompt‑based learning.

3.1 GPT‑1

12‑layer Transformer decoder with 768‑dimensional hidden states; trained with unsupervised pre‑training then task‑specific fine‑tuning. Limited generalization compared to later models.

3.2 GPT‑2

48‑layer decoder with 1600‑dimensional hidden states; trained on massive data without task‑specific fine‑tuning, enabling zero‑shot capabilities.

3.3 GPT‑3

175 billion parameters, 45 TB of training data, introduces in‑context learning (zero‑, one‑, few‑shot). Achieves strong performance on many NLP tasks but requires huge compute and exhibits hallucinations.

3.4 ChatGPT

Built on GPT‑3.5 with Reinforcement Learning from Human Feedback (RLHF). Provides more coherent dialogue, can write code, but still suffers from instability and limited reasoning.

3.5 GPT‑4

Multimodal model capable of processing images and up to 25 000 tokens of text. Improves reasoning, coding, and domain‑specific tasks while still facing high training costs and safety concerns.

4 Conclusion

Large language models have shifted NLP from rule‑based methods to pre‑trained, fine‑tuned systems that achieve state‑of‑the‑art results across many tasks. Future work must address data efficiency, interpretability, bias, hallucination, privacy, and computational cost.

5 References

A Neural Probabilistic Language Model

Recurrent Neural Network Regularization

Long Short‑Term Memory

Deep Contextual Word Embeddings

Attention Is All You Need

BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding

Improving Language Understanding by Generative Pre‑Training

Language Models are Unsupervised Multitask Learners

Language Models are Few‑shot Learners

Pre‑train, Prompt, and Predict: A Systematic Survey of Prompting Methods in NLP

Training language models to follow instructions with human feedback

Proximal Policy Optimization Algorithms

ChatGPT: Optimizing Language Models for Dialogue

Deep Residual Learning for Image Recognition