Data‑Centric AI Perspective on GPT Models: Training, Inference, and Maintenance

What Are Large Language Models and What Is the GPT Series?

Large language models (LLMs) are neural‑network language models that predict missing words based on context; they require massive amounts of data to learn general linguistic patterns. The GPT series (GPT‑1, GPT‑2, GPT‑3, InstructGPT, ChatGPT/GPT‑4) are built on the Transformer architecture and use attention mechanisms to model token relationships.

Subsequent GPT models share a similar structure to GPT‑1, differing mainly in scale (more layers, larger hidden dimensions).

What Is Data‑Centric AI?

Data‑centric AI, championed by Andrew Ng, is defined as “the discipline of systematically engineering the data used to build an AI system.” Unlike model‑centric approaches that focus on iterating models while keeping data fixed, Data‑centric AI emphasizes improving data quality and quantity.

Data‑centric AI is the discipline of systematically engineering the data used to build an AI system. — Andrew Ng

Data‑centric AI differs fundamentally from “data‑driven” methods, which still prioritize model development over data improvement.

The Data‑centric AI framework consists of three primary goals:

Training data development : building sufficient high‑quality data for model training.

Inference data development : constructing data used at inference time, e.g., adversarial test sets or prompts for prompt engineering.

Data maintenance : continuously ensuring data quality and reliability in production environments.

Why Data‑Centric AI Is a Key Reason for GPT Success

While increasing model parameters is important, OpenAI engineers invested heavily in improving data quality and scale. The article analyzes this through three dimensions.

Training data development :

GPT‑1 : used the BooksCorpus (≈4.6 GB raw text). No explicit Data‑centric strategies.

GPT‑2 : used WebText, collected from Reddit links with ≥3 karma, extracted with Dragnet and Newspaper, applied heuristic deduplication and cleaning.

GPT‑3 : used Common Crawl, filtered low‑quality documents via a classifier trained on WebText similarity, applied MinHashLSH deduplication, added BooksCorpus and Wikipedia.

InstructGPT : fine‑tuned with human‑generated answers; strict annotator selection, exams, and questionnaires ensured high‑quality labels.

ChatGPT/GPT‑4 : commercialized; continues to rely on massive, high‑quality data and RLHF.

Inference data development :

Modern GPT models are powerful enough that adjusting prompts (prompt engineering) can achieve many tasks without retraining. Prompt engineering, adversarial test sets, and soft‑prompt calibration are emerging research areas.

Data maintenance :

Continuous data collection from user interactions and feedback.

Data understanding tools for visualizing and analyzing user data.

Efficient data processing pipelines to handle the growing volume of user‑generated data.

What Can We Learn from the Success of Large Language Models?

Future predictions include:

Data‑centric AI will become increasingly important as model architecture matures.

LLMs will aid Data‑centric AI by generating high‑quality synthetic data and automating data cleaning.

The line between data and models will blur, with prompts acting as data containers and synthetic data feeding back into model training.

Despite the hype, the article emphasizes that improving data quality and quantity remains the most reliable way to boost AI performance.

https://github.com/davidjurgens/potato

For readers interested in deeper exploration, the article provides links to survey papers, short introductions, and a GitHub repository on Data‑centric AI.

最近面试BAT，整理一份面试资料《Java面试BAT通关手册》，覆盖了Java核心技术、JVM、Java并发、SSM、微服务、数据库、数据结构等等。
获取方式：点“在看”，关注公众号并回复 手册，领取，更多内容陆续奉上。