GPT vs BERT: How In-Context Learning Replaced Fine-Tuning in LLMs

1. What Is Generative Pre-Training?

The pre-training–fine-tuning paradigm, a cornerstone of deep learning success in computer vision, uses models like ResNet-50 pre-trained on millions of images to learn generic visual features (edges, textures, composition). For a specialized task such as tumor detection from brain CT scans with only a few thousand labeled examples, practitioners load the pre-trained ResNet-50 weights, replace the final fully connected layer with a two-class output, and fine-tune on the medical dataset, transferring and adapting the learned feature extraction to the new domain.

Natural language processing lacked an equivalent to ImageNet and easy universal annotation. The solution was unsupervised pre-training using an autoregressive language model whose objective is next-token prediction, allowing raw text to be used without human labels. Generative Pre-Training is therefore not a direct copy of vision pre-training but a paradigm centered on autoregressive generation, endowing models with unmatched content creation and sequence generation capabilities — the foundation of GPT's power as a dialogue agent and content tool.

2. GPT-1

Released in June 2018 via the paper "Improving Language Understanding by Generative Pre-Training", GPT-1 introduced the generative pre-trained transformer concept. It uses only the Transformer decoder, a Decoder-only architecture. The model is pre-trained on a large corpus and then fine-tuned on downstream tasks by adding a task-specific output layer.

3. BERT

Four months after GPT-1, Google published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (October 2018). BERT employs the Transformer encoder — an Encoder-only architecture — and uses bidirectional context, seeing both left and right context simultaneously during training. It introduces two novel pre-training tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In short, BERT's pre-training emphasizes understanding, while GPT's emphasizes generation.

4. GPT-2 and GPT-3: From Fine-Tuning to In-Context Learning

GPT-2 (2019, "Language Models are Unsupervised Multitask Learners") revolutionized the paradigm by eliminating fine-tuning. Instead, the pre-trained model directly processes downstream tasks guided by natural language prompts. GPT-3 (2020, "Language Models are Few-Shot Learners") scaled this approach to 175 billion parameters trained on 300 billion tokens. Crucially, GPT-3 formalized three in-context learning settings:

Zero-shot : Perform a task given only a natural language instruction.

One-shot : Provide a single example before asking the model to do a similar task.

Few-shot : Provide multiple examples before requesting the task.

Researchers found GPT-3's few-shot performance remarkable, sometimes surpassing supervised models that require fine-tuning.

References

1. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

2. https://www.ultralytics.com/zh/blog/what-is-resnet-50-and-what-is-its-relevance-in-computer-vision

3. https://arxiv.org/abs/1810.04805

4. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

5. https://arxiv.org/abs/2005.14165