Understanding Codex: Training Framework, Evaluation Methodology, and Model Performance in ChatGPT’s Code Generation Ability

In the ChatGPT technical analysis series, this third part focuses on the Codex model that endows ChatGPT with code‑writing capabilities. It outlines the overall design of Codex, including the collection of a 159 GB Python dataset from GitHub, a full pre‑training phase that produces a 12 B‑parameter model, and a supervised fine‑tuning stage that adds prompt‑completion pairs and unit tests sourced from algorithm‑competition sites and CI scripts.

The evaluation methodology departs from traditional BLEU scores and uses a specially constructed HumanEval benchmark consisting of function signatures, docstrings, implementations, and unit tests. The pass@k metric is introduced, measuring the probability that at least one correct answer appears among k sampled outputs, with detailed formulas and discussion of why larger k improves success rates.

During inference, Codex stops generation upon encountering tokens such as "\nclass", "\ndef", "\n#", "\nif", or "\nprint". Output sampling relies on nucleus sampling (top‑p = 0.95), which balances diversity and quality by truncating the probability mass.

The training details include a learning‑rate schedule (175‑step linear warm‑up followed by cosine decay), 100 B tokens, and Adam optimizer settings (β₁=0.9, β₂=0.95, ε=1e‑8, weight decay = 0.1). Special token embeddings are used to represent code‑specific symbols such as indentation and colons.

Performance results show that GPT‑3 achieves 0 % pass@1 on code tasks, while Codex (pre‑trained only) reaches 28.8 % and Codex‑S (pre‑trained + fine‑tuned) reaches 37.7 % pass@1. Further improvements are obtained by generating 100 candidates and selecting the one with highest mean log‑probability (44.5 %) or using oracle reranking via unit tests (77.5 %).

The article concludes that Codex’s ability to generate code stems from training on large Python corpora, supervised fine‑tuning with function‑level prompts, and evaluation with task‑specific metrics, and that larger models and more data promise continued gains.