Comprehensive Technical Overview of GPT Series, Transformers, and Emerging Capabilities in Large Language Models

Preface

Origin : From the hype around the Metaverse to the unprecedented impact of ChatGPT, the article aims to synthesize technical capabilities of large language models (LLMs) for a broad audience.

Elon Musk: "OpenAI was created as an open‑source, non‑profit counterweight to Google, but has become a closed‑source, profit‑driven company controlled by Microsoft."

The goal is to present a comprehensive technical capability report on LLMs.

What the Report Covers

Detailed GPT development timeline

Vision AIGC principles

Training models larger than 100B parameters

Prompt engineering

Perspectives on ChatGPT

Bill Gates: "The Age of AI has begun; AI is as revolutionary as mobile phones and the Internet."

Jensen Huang: "This is the iPhone moment for Artificial Intelligence."

Yann LeCun: "ChatGPT is not particularly innovative, and nothing revolutionary."

Geoffrey Hinton: "We are better at reasoning; we need to extract knowledge from far less data."

Content of This Article

How Transformers unified NLP and CV, becoming core to AIGC

Core technologies introduced in each GPT generation (1, 2, 3, 3.5, 4)

Pre‑training, supervised fine‑tuning (SFT), and reinforcement learning from human feedback (RLHF)

Complex reasoning and emergent abilities of large models

Challenges of training large models

From AIGC to AIGA (AI‑generated actions)

Large Language Models

Large Models

Large models bring emergent capabilities and are poised to become the foundational AI infrastructure.

Comparison between small and large models:

Data

Model

Training

Advantages

Small Model

Task‑specific annotated data

One model per task

Repeated task‑specific tuning

-

Large Model

Massive unlabeled data

Unified multimodal model

Few‑shot or fine‑tuning on small task data

Stronger performance, better generalization, lower cost

Language Models

Human language stores accumulated world knowledge, enabling inter‑generational knowledge transfer, which machines can process faster, continuously, and at scale.

LLMs improve the efficiency of knowledge creation, inheritance, and application.

Transformer

Originally CNN for vision and RNN for NLP, the Transformer now serves as a unified language for text, images, audio, and video.

Key components:

Auto‑Regressive modeling

Residual connections (ResNet) to alleviate gradient vanishing and weight matrix degeneration

Layer‑Norm (instead of Batch‑Norm) for stable training across variable sequence lengths

Masking in the decoder to prevent future token leakage

Scaled Dot‑Product Attention (with scaling by √d_k to avoid extreme softmax values)

Multi‑Head Attention for learning diverse patterns

Self‑Attention (Q=K=V) and its three variants: encoder self‑attention, decoder self‑attention (with mask), and encoder‑decoder cross‑attention

Positional Encoding to inject sequence order information

Parallel computation advantage: Transformers process the whole sequence simultaneously, unlike RNNs which depend on previous outputs.

Long‑Range Dependency

Transformers achieve a maximum path length of 1, whereas RNNs have a path length proportional to sequence length, leading to greater information loss in long sequences.

Transformer Evolution

GPT Series

GPT‑1

Introduced self‑supervised pre‑training on large text corpora followed by fine‑tuning on task‑specific data.

Self‑supervised pre‑training

Unsupervised pre‑training

Contrastive pre‑training

Challenges: designing a unified loss and transferring learned knowledge to downstream tasks.

Masked language modeling predicts the next token; BERT masks interior tokens for fill‑in‑the‑blank tasks.

GPT‑2

Key innovation: zero‑shot capability—no task‑specific labels or fine‑tuning required; prompts guide the model.

GPT‑3

Scale increased to 175 B parameters; introduced in‑context learning (few‑shot prompting) without gradient updates.

InstructGPT

Three‑stage learning:

Unsupervised pre‑training (large text corpus)

Supervised fine‑tuning (SFT) with high‑quality dialogue examples

Reward modeling & PPO (RLHF) to align with human preferences

Model variants:

SFT → text-davinci-002

RLHF → text-davinci-003 (restores context learning while improving zero‑shot performance)

GPT‑4

Introduces multimodal capabilities via Vision Transformer (ViT) and masked patch prediction.

ViT splits images into 16×16 patches, treats each patch as a token, and processes them with a Transformer encoder.

Despite lacking CNN‑style inductive biases (locality, translation equivariance), ViT outperforms CNNs when trained on massive datasets.

Emergent Abilities

Defined as qualitative behavioral changes resulting from quantitative system changes.

In‑Context Learning (few‑shot prompting)

Chain‑of‑Thought (step‑by‑step reasoning prompts)

Challenges

Data

High‑quality SFT data reduces reliance on RLHF. GPT‑3’s data pipeline involved filtering, deduplication (LSH), and augmentation with curated datasets.

Predictable Scaling

OpenAI’s ability to forecast large‑model performance from small‑scale experiments is termed "predictable scaling".

From AIGC to AIGA

AIGA (AI‑generated actions) extends generative AI to decision‑making by translating natural language into formal APIs or executable commands for interaction with environments.

Typical pipeline: Natural language → Formal language/API → Executable action.

References

How does GPT Obtain its Ability? Tracing Emergent Abilities of LLM – Yaofu