Understanding the Principles Behind ChatGPT: NLP, Transformers, and Reinforcement Learning

Preface

Many have experienced the impressive abilities of ChatGPT in writing, answering, coding, translating, and more. Curious readers often wonder how it can be "all‑knowing" and perform diverse tasks such as acting as a Linux terminal or a front‑end interview examiner. This article starts from NLP principles to reveal the underlying technology of ChatGPT.

NLP Technology

Natural Language Processing (NLP) is an AI field that enables computers to understand, analyze, process, generate, and converse in human language.

Human‑machine interaction optimization: extracting key information from input text for downstream applications (e.g., voice‑controlled devices).

Generative tasks: understanding user input and producing the desired information (e.g., Q&A, code generation).

Translation: converting one language to another while preserving naturalness.

Information summarization and aggregation: automatic classification and recommendation in content feeds.

ChatGPT integrates many of these NLP capabilities, offering a user‑friendly product that has shifted research focus toward large‑scale generative models.

Generative Language Models

State‑of‑the‑art large language models such as GPT or BERT can be viewed as deep probabilistic models of word sequences. When generating text, the model predicts the next token based on preceding tokens. The figure below illustrates a simple example where the model completes the input "你好" by selecting the most probable next characters.

Similar probability‑based completions appear in search‑engine suggestion boxes and input‑method candidate lists, as shown in the following images.

Mathematically, the model maximizes the conditional probability p(w_i | w_{i‑1}, …, w_{i‑n}) for each token, often using log‑probabilities to avoid underflow. This objective is equivalent to maximum‑likelihood estimation.

Deep Learning Enables GPT to Acquire Language Skills

ChatGPT is built on deep learning, which automatically learns syntactic structures from massive text corpora and can even capture abstract syntax trees (AST) for code.

What Is Deep Learning?

Deep learning is a class of machine‑learning algorithms based on artificial neural networks with many layers, excelling at image, speech, and language tasks.

Its core idea is end‑to‑end feature extraction and parameter optimization to maximize predictive accuracy.

Applications span computer vision, speech recognition, NLP, and more.

For language models, deep learning allows the network to learn grammar, programming language structures, and world knowledge.

Decoding the GPT Acronym

Generative : GPT is a unidirectional (autoregressive) language model that predicts the next token from previous context. Unlike bidirectional models such as BERT, GPT focuses on generation.

Pre‑trained : Pre‑training endows the model with general knowledge. While BERT typically requires downstream fine‑tuning, GPT‑3‑scale models exhibit strong few‑shot and zero‑shot abilities without additional parameter updates.

Transformer : The Transformer architecture, introduced by Google in 2017, relies on self‑attention to capture long‑range dependencies more efficiently than RNNs.

The following table summarizes the evolution of GPT versions:

Version

Features

Parameter Scale

GPT‑1

Initial decoder‑only Transformer; unsupervised + supervised training; fine‑tunable downstream.

117 M parameters, 5 GB data

GPT‑2

Decoder‑only; enhanced unsupervised learning; introduced few‑shot capability.

1.5 B parameters, 40 GB data

GPT‑3

Scaled up decoder‑only model; massive training data.

175 B parameters, 45 TB data

GPT‑3.5 (ChatGPT)

Added dialogue and code data; incorporated InstructGPT reinforcement learning.

175 B parameters, 45 TB data

Zero‑Shot, One‑Shot, and Few‑Shot Learning

Traditional models require fine‑tuning on each downstream task. GPT introduced "few‑shot" learning, where a handful of examples are provided in the prompt (In‑Context Learning) and the model performs the task without updating its parameters. Zero‑shot learning supplies no examples, relying solely on the model's pre‑trained knowledge.

Reinforcement Learning Aligns GPT Outputs

What Is Reinforcement Learning?

RL is a branch of machine learning where an agent interacts with an environment, receives observations and rewards, and learns a policy to maximize cumulative reward.

It has been applied to games, robotics, autonomous driving, speech recognition, and more.

ChatGPT’s behavior is refined through a three‑step RL pipeline:

Supervised Fine‑Tune (SFT) : Human‑written prompts and responses are collected to train an initial model.

Reward Model (RM) Training : The model generates answers to sampled prompts; humans rank them, and a reward model is trained on these rankings.

Proximal Policy Optimization (PPO) : The reward model provides feedback to further fine‑tune the SFT model via RL.

These steps enable the model to prefer appropriate answers (e.g., giving the correct location of Shanghai’s tallest building) and to reject harmful or non‑compliant content.

Transformer and Attention Mechanism

GPT uses the decoder part of the Transformer architecture. Transformers, introduced in the paper "Attention Is All You Need," replace recurrent networks with self‑attention, allowing parallel computation and better handling of long‑range dependencies.

RNN Overview

Recurrent Neural Networks process sequential data by maintaining a hidden state matrix A that accumulates information from previous tokens. However, RNNs suffer from forgetting long‑range context, making them less suitable for long text generation.

Attention Mechanism

Attention addresses the weak coupling between encoder and decoder states in Seq2Seq models. By computing Query‑Key‑Value interactions, the decoder can focus on relevant encoder states for each output token.

Self‑Attention extends this idea by applying Q, K, V within the same sequence, enabling each token to attend to all others.

Transformer Model

A Transformer consists of stacked attention blocks. Each block contains multi‑head self‑attention followed by a feed‑forward layer. Encoder blocks have only self‑attention; decoder blocks add an extra cross‑attention layer to incorporate encoder outputs.

BERT vs. GPT

Both are Transformer‑based, but BERT uses the encoder for masked language modeling (bidirectional), while GPT uses the decoder for autoregressive generation (unidirectional). BERT excels at understanding and classification; GPT excels at generation and few‑shot adaptation.

Conclusion

ChatGPT builds on decades‑old Transformer and attention research, leveraging modern GPU power, massive decoder‑only pre‑training, and reinforcement learning to produce a versatile large language model that points toward the future of general artificial intelligence.

References

王树森 NLP 入门: https://www.youtube.com/@ShusenWang

Attention paper: https://arxiv.org/abs/1706.03762

GPT few‑shot paper: https://arxiv.org/abs/2005.14165

InstructGPT: https://arxiv.org/pdf/2203.02155.pdf

Illustrated Transformer blog: http://jalammar.github.io/illustrated-transformer/

Additional Chinese article: https://zhuanlan.zhihu.com/p/48508221