Analysis of Large Language Models: Capabilities, Training Methods, and Limitations – Summary of Prof. Qiu Xipeng’s Lecture

On February 17, 2023, the Gaoling AI Institute hosted its first academic report, featuring Prof. Qiu Xipeng from Fudan University, who delivered a talk titled “Analysis of Large Language Model Capabilities and Applications.” Over 500 students and faculty attended in person and online.

The lecture began with a brief history of ChatGPT and its predecessors, tracing the evolution from the 2017 Transformer framework to GPT‑3, CodeX, WebGPT, InstructGPT, and finally ChatGPT. It highlighted how the Transformer architecture enables scaling to billions of parameters, leading to emergent abilities that break the traditional scaling law.

Three major technical advances of ChatGPT were discussed:

In‑context learning : providing task descriptions and a few examples in the prompt allows the model to perform few‑shot learning without parameter updates.

Chain‑of‑Thought (CoT) : prompting the model with step‑by‑step reasoning chains improves accuracy on complex logical problems.

Natural‑instruction learning : adding a natural‑language instruction before the input aligns downstream tasks with the model’s generative capabilities.

The training pipeline follows the RLHF (Reinforcement Learning from Human Feedback) paradigm used in InstructGPT, with three stages: supervised fine‑tuning on high‑quality instruction‑output pairs, reward‑model training on ranked model outputs, and PPO‑based reinforcement learning to produce the final ChatGPT.

Capability assessment was organized into four dimensions—Know Knowns, Know Unknowns, Unknow Knowns, and Unknow Unknowns—illustrating how scaling and CoT unlock previously hidden knowledge. Evaluation methods such as HELM and the GAOKAO‑Bench (Chinese college‑entrance exam) were presented, showing ChatGPT’s performance comparable to a 500‑point test‑taker.

The speaker also addressed limitations: current form is text‑only, controllability is limited, reasoning can still be weak, and the model lacks real‑world grounding. Potential solutions include multimodal front‑ends, more extensive human alignment, and integration with tools like WebGPT.

Two audience Q&A excerpts were included, covering privacy concerns in LLMs and the possibility of self‑play training akin to AlphaZero, both emphasizing the need for trustworthy AI.