Interview Guide: Core Differences Between PPO and GRPO Algorithms for Large Model Fine‑Tuning

Problem Context

Interview questions on large‑model fine‑tuning often require a concise description of the reinforcement‑learning algorithms PPO and GRPO.

Proximal Policy Optimization (PPO)

PPO follows an actor‑critic architecture:

Policy network (Actor) – outputs an action distribution for a given state.

Value network (Critic) – estimates the state value.

During training the actor samples an action, the critic predicts its value, and a reward function scores the action. The advantage is computed as A = Reward - Value. Positive advantage increases the action’s probability; negative advantage decreases it, aligning the policy with human preferences.

Example:

Action: "你好，好久不见"
Value (Critic) = 0.6
Reward = 0.5
A = 0.5 - 0.6 = -0.1  (negative → probability reduced)

Action: "你好，好久不见，最近过得怎么样"
Value = 0.65
Reward = 0.8
A = 0.8 - 0.65 = 0.15  (positive → probability increased)

The advantage guides the policy update so that outputs preferred by the reward model become more likely.

Group Relative Policy Optimization (GRPO)

GRPO, introduced by DeepSeek in the 2024 paper DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , removes the critic entirely. For each state it samples a set of k actions, computes raw rewards for each, and normalizes the rewards relative to the group to obtain an advantage used for policy updates.

Illustrative walk‑through (group size 3):

Sampled answers: A, B, C
Rewards: r_A = 85, r_B = 92, r_C = 70
Group mean = (85+92+70)/3 = 82.33
Advantage_A = 85 - 82.33 = +2.67 (positive)
Advantage_B = 92 - 82.33 = +9.67 (positive)
Advantage_C = 70 - 82.33 = -12.33 (negative)

Positive advantages reinforce answers A and B; the negative advantage suppresses C. Because no value network is trained, the pipeline is lighter and training stability improves when the reward model is reliable.

Why Many Models Prefer GRPO Over PPO

Eliminating the critic reduces memory and compute overhead.

Large language models can readily generate multiple candidates for the same prompt, satisfying the group‑wise comparison requirement and increasing training throughput.

GRPO has shown strong performance on mathematical and programming tasks, which are current focus areas for large‑model development.

Primary Risk of GRPO

GRPO relies on sampling enough candidates per prompt and on a reward model with sufficient discriminative power. Insufficient samples or a weak reward model can produce sparse training signals or over‑fitting. The absence of a global baseline (the critic) may also cause instability on long‑horizon tasks.

Choosing the Group Size k in Practice

Typical values are k = 4–8. Smaller k weakens the normalization effect; larger k increases memory and compute consumption. When the reward model is stable, a smaller k suffices; with noisy rewards, a larger k improves training stability.