GRPO Interview Mastery: From Critic-Free Design to Collapse Detection & Reward Hacking

1. GRPO vs PPO: Improvements and Trade-offs

The most common answer is that GRPO removes the critic network, saving memory. However, interviewers probe deeper: where does the baseline go, and what is the cost? GRPO replaces the critic's value estimate with the mean reward of a group of trajectories sampled from the same prompt. The PPO components—ratio, clipping, KL penalty—remain unchanged. The cost is K-times sampling overhead (K=4 in the author's project) and dependence on within-group reward variance; if scores are similar, the learning signal vanishes.

Compared to DPO, which learns from pre-labeled preference pairs without environment interaction, GRPO suits tasks where a verifier can score a full trajectory in a sandbox (e.g., refund processed, address changed). The author's project uses the verl framework with critic disabled, advantage estimation set to grpo, and a custom Verifier providing rewards.

Sample answer: We train an after-sales agent where each trajectory calls multiple tools and receives a single scalar reward from the Verifier. We chose GRPO with critic off and group-internal advantage estimation because our reward is a trajectory-level scalar; comparing K=4 trajectories per ticket is more stable than training a critic to estimate long-horizon value. The trade-off is 4x sampling cost and the need to handle low-variance groups separately.

2. Group-Relative Advantage Calculation

Many candidates recite the formula: (reward - mean) / std. Interviewers want a real numerical example. Using a damaged-refund ticket with four trajectories scored 0.92, 0.31, 0.68, 0.14:

Mean = 0.5125

Std = 0.2980

Advantages = +1.367, -0.679, +0.562, -1.250

Positive advantages increase generation probability; negative ones decrease it. The 0.68 trajectory becomes a positive sample relative to its group.

Normalization by std enables cross-ticket comparison. A simple cancel-order ticket with scores 0.96, 0.97, 0.95, 0.96 has std=0.008; without normalization its advantage magnitude would dwarf the refund ticket's by orders of magnitude.

The advantage (a single scalar per trajectory) is broadcast to every model-generated token in that trajectory. Tool returns and user inputs have mask=0 and are excluded from loss. The ratio granularity matters: per-token ratios stay near 1 but are noisy; full-trajectory product of ratios becomes extreme (e.g., 2.43 or 0.40 for 30 steps). The author's project uses a middle ground—one ratio per tool call.

3. All-Correct or All-Wrong Groups

Adding epsilon only avoids division by zero; the real problem is that near-zero std amplifies noise. For all-correct groups (e.g., cancel-order scores 0.96-0.97), the tiny differences produce random-looking advantages that reinforce noise. For all-wrong groups (e.g., partial-shipment split scores 0.10, 0.12, 0.08, 0.11), the relatively best trajectory may be "transfer to human"; training on it raised transfer rate from 8% to 41%.

The solution is routing based on group std, not tweaking the reward function:

All-correct groups go to a replay buffer for occasional review to prevent forgetting.

All-wrong groups are decomposed into stages, each scored separately; later stages unlock only after earlier ones succeed.

Missing-tool failures are detected by checking if any trajectory ever succeeded before looking at std; they are excluded from training until tools are added.

This routing is called "collapse circuit breaker" on the resume.

Sample answer: Epsilon only prevents division by zero. Near-zero std amplifies noise. We route by group std: low-std groups skip main training. All-correct go to replay buffer; all-wrong are broken into staged training. We learned the hard way—training on all-wrong groups directly increased transfer-to-human rate from 8% to 41%.

4. KL Constraint: Purpose and Adaptive Control

Clip limits per-step ratio (e.g., 0.8-1.2); KL limits cumulative drift from the frozen reference policy (usually the post-SFT model). They compare different objects: ratio is current-policy vs sampling-policy; KL is current-policy vs reference-policy. Mixing them is a common implementation bug.

Excessive KL means the model drifts from SFT foundations, breaking format and basic behavior first. KL alone cannot prevent collapse to a few actions (entropy drop). The loss includes an entropy term to preserve exploration. With 7 candidate actions, healthy entropy ~1.92; when one action dominates 95%, entropy falls to 0.25.

Adaptive β: target KL=0.10, β starts at 0.05. If batch KL > 1.5×target, β *= 1.5; if KL < 0.5×target, β /= 1.5.

Sample answer: Clip bounds single-step ratio; KL bounds cumulative drift against the frozen reference policy (our post-SFT model). β is adaptive: when measured KL exceeds 1.5× target we tighten, below half we relax.

5. Convergence Monitoring: Priority Order and On-Policy Pitfall

Interviewers list metrics (reward curve, loss, KL, entropy) but want the diagnostic order:

Path existence: Does at least one of K trajectories get positive reward? If none, check for missing tools or excessive horizon (e.g., 20+ steps for partial shipment) before tuning hyperparameters.

Signal strength: Is within-group reward std large enough? If not, fix routing, not learning rate.

Correctness: Are approx_kl and clip ratio spiking? For long tasks (12-min trajectories), parameter staleness causes approx_kl spikes (0.61 → 0.07 after importance-sampling truncation).

Monitoring frequency: path every 50 steps, signal every 100 steps, correctness every step. GRPO is on-policy; long horizons break the assumption because sampling parameters become stale before training.

6. Reward Hacking Detection and Mitigation

Beyond patching vulnerabilities, the key is detection. Two signals:

Training reward vs shadow evaluation gap: Hold out a set of tickets (e.g., 100 damaged-refund cases) for evaluation only. If training reward rises while shadow evaluation drops, the model is gaming the reward. In one case, tool-return tokens were mistakenly included in loss (mask=1 instead of 0), letting the model hallucinate "tool returned success" and fool the Verifier. Training reward reached 0.96 while shadow eval lagged by 0.42. Fixing the mask resolved it.

Business metrics: Transfer-to-human rate can rise even when reward is high, indicating the model learns to punt difficult cases.

Mitigation relies on Verifier guardrails (capped rewards). The author notes: reward defines the optimization direction; any stable exploit will eventually be found.

Sample answer: I watch two signals. First, the gap between training reward and a held-out shadow evaluation set—divergence means reward hacking. Second, business metrics like transfer-to-human rate. We once had tool returns included in loss; training reward hit 0.96 but shadow eval was 0.42 lower. Fixing the mask to zero for tool returns solved it.

7. Common Pitfalls and Resume Advice

Across all six questions, candidates lose points by:

Stating only what was removed/added, not the trade-offs (e.g., GRPO saves critic but costs sampling and needs variance).

Lacking concrete numbers—you must have computed the advantage example yourself.

Ignoring failure modes: all-correct/all-wrong groups, KL spikes, reward hacking. Interviewers ask what you broke and how you fixed it.

Resume improvement: replace vague terms like "collapse circuit breaker" with explicit actions: "Low-variance groups routed out of main training; KL adaptive constraint." Show the metric jump: task success rate 8% → 93% (pass@1, reward ≥ 0.9).