Skip to content
HKU NeurIPS 2026: Implicit Advantage Symmetry Explains GRPO's Dual Bottlenecks

HKU NeurIPS 2026: Implicit Advantage Symmetry Explains GRPO's Dual Bottlenecks

Hong Kong University researchers identify implicit advantage symmetry as the root cause of GRPO's poor exploration and difficulty adaptation, proposing A-GRAE to dynamically break symmetry and achieve simultaneous gains in Pass@1 accuracy and Pass@k diversity across math and multimodal benchmarks without extra compute.

Machine Heart
Machine Heart
Machine Heart

With the rise of DeepSeek-R1, GRPO (Group Relative Policy Optimization) has become a standard paradigm for LLM post-training. However, recent research highlights two fundamental flaws: (1) GRPO fails to expand the model's capability boundary, merely trading off exploration and exploitation within existing skills, and even under large sampling budgets (e.g., Pass@256) it can underperform the base model; (2) GRPO's reward mechanism ignores actual problem difficulty and the model's current mastery, leading to inefficient training.

Professor Qu Jingqiong's team at HKU introduces a unified theoretical lens — Implicit Advantage Symmetry — to explain both defects. In GRPO, the group-relative advantage estimation normalizes rewards by subtracting the group mean and dividing by the standard deviation. This single step simultaneously governs intra-group reward/punishment and inter-sample learning signal allocation. The team proves this normalization hides a symmetry that stifles evolution.

Group-Level Symmetry

Within a group, the sum of absolute advantages for correct and incorrect trajectories is strictly equal (zero-sum property). Theorem 1 shows that under standard GRPO (C=0), only sampled trajectories receive direct logit updates; unsampled trajectories — including low-probability correct paths — get no direct gradient signal. Consequently, the model cannot learn new solution paths it hasn't already sampled, suppressing exploration.

Sample-Level Symmetry

Theorem 2 derives that the total absolute advantage allocated to a prompt is proportional to √[p(1-p)], where p is the empirical success rate. This peaks at p=0.5 (medium difficulty), while easy (p=0.75) and hard (p=0.25) samples receive identical weight. This static symmetry ignores dynamic distribution shift: as the model improves, easy samples dominate the pool, causing overfitting on trivial data and starving hard samples of learning signal.

Controlled Experiments: Breaking Symmetry

The team designed ablation experiments on Qwen2.5-Math-7B to isolate causal effects.

Experiment I (Group-Level): Amplifying correct-trajectory weights (Positive-Dominant) causes severe entropy collapse and worst large-budget performance; suppressing them (Negative-Dominant) greatly boosts exploration but risks late-training instability. Takeaway: Suppressing correct-path weights aids exploration but requires a dynamic decay mechanism.

Experiment II (Sample-Level): No single static difficulty weighting is universally optimal; the best strategy depends on benchmark and training stage. Early focus on easy samples helps acquire reasoning formats; later shift to hard samples pushes the performance ceiling. Takeaway: A non-stationary easy-to-hard dynamic curriculum outperforms static or hard-only strategies.

Unified Perspective on Existing Methods

The advantage-symmetry framework unifies previously fragmented GRPO improvements:

Negative-reinforcement and Pass@k-oriented methods implicitly break group-level symmetry by down-weighting over-learning on correct paths to foster exploration.

High-entropy token or hard-sample emphasis methods break sample-level symmetry by reallocating learning signals toward uncertain or difficult regions.

A-GRAE: Asymmetric Group Relative Advantage Estimation

To jointly address both symmetries without extra compute, the team proposes A-GRAE, a plug-and-play module that reshapes advantages after initial computation.

Sample-Level Dynamic Attention Shift: Using batch-wise mean reward R̄ as a proxy for model capability, A-GRAE applies a difficulty-dependent weighting: w(p) = (1 - R̄) * (1 - p) + R̄ * p where p is the sampling success rate. As R̄ grows, weight shifts from the easy-focused term (1-p) to the hard-focused term p, realizing an adaptive easy-to-hard curriculum.

Group-Level Attenuation Suppression: For correct responses, advantages are scaled by a decaying factor: A' = A * exp(-α * R̄) with α>0. Early training (low R̄) preserves exploration incentives; late training (high R̄) restores stability.

Experimental Validation

Evaluated on 7 benchmarks (AIME2025, AMC23, MATH, Geo3K, MathVision, etc.) with Qwen2.5-Math-7B, DeepSeek-R1-7B, and Qwen2.5-VL-3B.

Consistent gains: A-GRAE improves GRPO and variants (e.g., DAPO) across text and multimodal tasks. On AIME2025, DAPO+A-GRAE averages 53.3% (k=1..256) vs. 52.2% for vanilla DAPO.

Accuracy-diversity win-win: Unlike prior methods that trade Pass@1 for Pass@k, A-GRAE lifts both simultaneously.

Strong generalization: Significant improvements on multimodal medical reasoning and OOD tasks demonstrate broad applicability.

The paper (NeurIPS 2026) and code are open-sourced at https://github.com/HKU-HealthAI/A-GRAE.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

reinforcement learningGRPOexplorationLLM post-trainingNeurIPS 2026A-GRAEdifficulty adaptationimplicit advantage symmetry
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.