Technical Principles and Training Process of ChatGPT

ChatGPT has become a hot topic in the AI community, sparking widespread discussion and testing examples due to its impressive performance.

The article examines the technical mechanisms behind ChatGPT, asking whether its capabilities could replace existing search engines like Google or Baidu.

ChatGPT builds on the large language model GPT‑3.5 and employs a three‑stage pipeline: (1) supervised fine‑tuning with human‑annotated data to teach the model to understand diverse prompts; (2) training a reward model (RM) using pair‑wise ranking of multiple model outputs; (3) reinforcement learning (PPO) that uses the RM to assign scores and update the language model.

Iterating between stages two and three progressively improves the model, as the RM provides high‑quality pseudo‑labels for further training.

The author argues that, despite its strengths, ChatGPT cannot yet replace traditional search engines because of hallucinations, difficulty incorporating new knowledge quickly, and high computational costs.

Potential solutions include integrating retrieval‑augmented generation, leveraging systems like Sparrow and LaMDA for evidence display and up‑to‑date knowledge, and separating reward models for helpfulness and safety.

A proposed next‑generation search architecture combines a conventional search engine (as a supporting engine) with ChatGPT as the primary engine, using the search engine to verify answers and supply fresh information.

The article concludes with a disclaimer that the views expressed are personal and that the content is for learning and discussion only.