Uncovering ChatGPT’s Emergent Abilities: A Technical Roadmap from GPT‑3 to GPT‑3.5

Recent breakthroughs of OpenAI’s ChatGPT have astonished the AI community, prompting the question of how these models became so powerful. This article dissects the emergent abilities of ChatGPT, reconstructs the technical roadmap from GPT‑3.5 series to the current model, and aims to provide a transparent guide for reproducing GPT‑3.5 in the open‑source community.

Why ChatGPT Is So Powerful

ChatGPT impresses with strong language generation, instruction following, and code writing, far exceeding early expectations for large language models.

2020 GPT‑3 and Large‑Scale Pre‑training

Language generation: Predicts the next token given a prompt.

In‑context learning: Solves new tasks by conditioning on a few examples without parameter updates.

World knowledge: Stores factual and commonsense information from massive corpora (≈3 trillion tokens, 175 billion parameters).

These capabilities stem directly from the massive pre‑training dataset and model size.

From GPT‑3 to ChatGPT (2020‑2022)

OpenAI released a series of models: the original GPT‑3 (davinci) in July 2020, Codex in July 2021, supervised instruction‑tuned models (davinci‑instruct‑beta, text‑davinci‑001) in 2021, and finally the instruction‑tuned code‑davinci‑002, text‑davinci‑002, text‑davinci‑003, and ChatGPT between March and November 2022.

Instruction tuning unlocks the ability to follow human directives, while code training injects programming knowledge.

Code‑Davinci‑002 and Text‑Davinci‑002: Code Training and Instruction Tuning

Code‑davinci‑002 is the base model trained on large code corpora; text‑davinci‑002 results from supervised instruction tuning of that base. The former excels at in‑context learning, the latter at zero‑shot tasks.

Both models inherit the three core abilities (generation, world knowledge, in‑context learning) and add code understanding.

Complex Reasoning via Chain‑of‑Thought

Chain‑of‑thought reasoning, weak in the original GPT‑3, becomes strong in code‑davinci‑002 and text‑davinci‑002, likely as a by‑product of code training. Evidence includes superior performance on GSM8K and other math benchmarks.

Instruction Tuning vs RLHF

All three later models (text‑davinci‑002, text‑davinci‑003, ChatGPT) undergo instruction tuning. Text‑davinci‑002 uses supervised tuning, while text‑davinci‑003 and ChatGPT incorporate reinforcement learning from human feedback (RLHF), which makes responses longer, more balanced, and better at refusing out‑of‑scope queries.

RLHF does not inject new abilities; it unlocks existing ones and aligns the model with human preferences, at the cost of some performance (the “alignment tax”).

What GPT‑3.5 Still Cannot Do

Rapidly revise its own beliefs when presented with contradictory evidence.

Perform strict formal reasoning required for exact mathematical proofs or first‑order logic.

Directly retrieve up‑to‑date information from the internet (though internal research such as WebGPT explores this).

Conclusion

GPT‑3.5’s strengths—language generation, world knowledge, instruction following, code understanding, and RLHF‑driven alignment—originate from large‑scale pre‑training, extensive instruction datasets, and code‑centric training. Code‑davinci‑002 appears to combine all major abilities, while subsequent instruction‑tuned variants trade off some capabilities for safety and conversational quality. The roadmap presented here should help the open‑source community reproduce and extend these models.