How DeepSeek‑R1’s Reinforcement Learning Earned a Nature Cover

Overview

DeepSeek‑R1 was featured on the cover of Nature after its January 2024 paper, “Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” demonstrated that pure reinforcement learning (RL) can unlock unlimited reasoning ability in large language models (LLMs). Reviewers praised the model’s openness, safety, and the novelty of a peer‑reviewed mainstream LLM.

Key Achievements

After open‑source release, the model became the most downloaded LLM on Hugging Face with over 10.9 million downloads.

Training cost was disclosed as $294 000 (plus roughly $6 million for the base model), dramatically lower than the costs reported by OpenAI or Google.

Performance on the AIME 2024 benchmark rose from 15.6 % to 77.9 % pass@1, reaching 86.7 % with self‑consistency decoding, surpassing human averages.

Training Framework

The team replaced the traditional supervised‑fine‑tuning (SFT) stage with a minimalist RL pipeline that only requires two elements in each task:

Task format: the answer must contain a <think> section for the reasoning process and a <answer> section for the final answer.

Reward signal: a binary reward based solely on the correctness of the final answer, regardless of the reasoning path.

This “bare‑bones” approach allowed the model to undergo rapid “wild growth” in reasoning capability.

GRPO Algorithm

Instead of the resource‑intensive PPO algorithm, DeepSeek adopted Group Relative Policy Optimization (GRPO). GRPO generates a set of answers (e.g., 16) for each question, computes each answer’s advantage relative to the group average, and weights the policy update accordingly. This reduces computation while preserving stability.

Reward Design

Two‑track rewards were employed:

Rule‑based rewards for math, programming, and logic tasks, enforcing strict accuracy and correct <think> formatting.

Model‑based rewards for general tasks, including a usefulness model (evaluates relevance and utility) and a safety model (detects harmful or biased content). The usefulness model only judges the final summary, leaving the reasoning process free.

Multi‑Stage Training Pipeline

The training proceeded through four stages:

Cold start : fine‑tune on thousands of high‑quality dialogue data to improve language fluency.

First RL round : focus on reasoning tasks with rule‑based rewards.

Large‑scale SFT : mix reasoning data with massive non‑reasoning data (writing, QA, code) to broaden general capabilities.

Second RL round : apply both rule‑based and model‑based rewards, lowering the sampling temperature to 0.7 and introducing the usefulness and safety models only in the final 400 steps to avoid reward hacking.

Key hyper‑parameters: learning rate 3×10⁻⁶, KL coefficient 0.001, GRPO clip ε = 10, inference temperature = 1 (first stage) then 0.7, batch size 512, 32 questions per step, reference model update every 400 steps.

Language Consistency Reward

During the first RL stage the model frequently mixed Chinese and English within the <think> section. A language‑consistency reward was added to encourage higher Chinese token ratios for Chinese prompts, improving readability at a negligible performance cost.

Challenges & Future Directions

Limited structured output and tool use (e.g., calculators, search engines).

Sensitivity to prompt design; excels in zero‑shot settings but struggles with few‑shot prompting.

Risk of reward hacking, especially for subjective tasks like poetry generation.

OpenAI accusations of data leakage, which DeepSeek refuted while acknowledging that the base model was pre‑trained on publicly available web data.

Reviewers from Nature and Hugging Face consider the approach a “revolution” that may inspire broader adoption of pure RL for LLM reasoning.

References

Nature paper: https://www.nature.com/articles/s41586-025-09422 Nature commentary: https://www.nature.com/articles/d41586-025-03015-6