Agentic RL: Cutting‑Edge Techniques from GLM‑5.2 and Qwen

GLM‑5.2: From GRPO to PPO

The GLM‑5.2 team introduced slime , a unified training‑inference platform that supports white‑box rollout, black‑box rollout, compact trajectory, and sub‑agent workflow. For long‑chain tasks, they replaced group‑wise optimization with a critic‑based PPO that estimates token‑level advantage, eliminating the need for intra‑group ranking and naturally handling compacted sub‑trajectories.

An anti‑hack module monitors each tool call online. When suspicious actions (e.g., reading protected files, copying from reference answers, pulling source from GitHub, or chaining commands to read hidden files) are detected, the call is intercepted and replaced with fabricated output instead of aborting the whole rollout, keeping training signals stable.

Qwen: The Verification Horizon

Qwen’s paper (arXiv:2606.26300v1) argues that a single reward mechanism cannot forever solve coding agents; verification must co‑evolve with the model. Three verification dimensions are defined:

Scalability – low‑cost, large‑scale reward provision.

Fidelity – closeness to true human intent.

Robustness – resistance to adversarial inputs and model‑optimization pressure.

Four verification strategies are evaluated:

Unit tests – cheap but narrow.

LLM judge – broader semantic coverage but vulnerable to stronger models.

Human review – most faithful but not scalable.

1. General Software‑Engineering (SWE) Tasks

The team builds on the SWE‑Universe pipeline, extracting real GitHub pull‑request patches and test patches, executing them in Docker with evaluation.sh. To improve fidelity, they add an Agentic Quality Judge that inspects code, files, and test cases, outputting two binary signals: instruction_clear and instruction_ut_align. Cleaning the data with this judge removes many “hard” tasks that were actually low‑quality, boosting RL sampling efficiency and raising scores on several SWE benchmarks.

Reward‑hacking is mitigated by a two‑stage monitor: first a rule‑based filter for obvious cheats, then an LLM judge that decides whether the action is a legitimate solution step or a shortcut. After deployment, the proportion of hacked solutions dropped from 28.57 % to 0.56 % while clean solutions rose from 40.22 % to 60.53 %.

2. Front‑End Development Tasks

Static scoring suffers from subjectivity and reward‑hacking (e.g., inflating CSS/JS length). The authors introduce a Rubric‑based Static Judge with 25.9 checklist items covering functionality (37.7 %), content (19.0 %), visual effect (13.3 %), layout (12.9 %), UX (9.3 %), and technical specs (7.2 %). Kendall’s τ ≥ 0.93 shows high consistency across prompts.

To capture dynamic behavior, a three‑stage Interactive Agentic Judge is built:

Action Planner generates a full action sequence (e.g., click → press → fill).

Render Server runs the sequence with Playwright, recording screenshots, DOM changes, and console output.

Judge Model scores the recorded trajectory against the rubric.

Experiments show that static judges still allow reward‑hacking (code length explodes with no score gain), whereas the interactive judge eliminates this by requiring functional UI responses. The approach lifted Qwen‑Max’s CodeArena front‑end rank to fourth place.

3. Real‑World Tasks – Implicit User Feedback

Most agents train on sandbox test cases, which diverge from open‑ended real‑world requests. The authors treat user utterances as implicit feedback: a silent continuation implies approval, explicit “not correct” signals rejection. Using a large internal dataset (≈125 k trajectories, 535 k annotated rounds), they observe:

76.6 % neutral, 20.0 % negative, 3.5 % positive feedback.

Negative feedback is high‑confidence (81.8 %); main failure modes are execution errors (56.6 %) and misunderstanding intent (21.1 %).

Discarding negative samples harms performance (score drops 41.8 % → 37.2 %).

They train a Span‑Level KTO model that treats each dialogue span as a preference unit, weighting good spans up and bad spans down while preserving neutral data for language modeling. This yields higher solve rates, lower “brain‑retry” ratios (‑34.5 %), and better explanatory ability (‑26.5 %).

GenAC: Generative Critics for Value Modeling

Traditional discriminative critics suffer from two flaws: lack of scalability (MSE does not drop when scaling from 0.6 B to 14 B) and extreme seed‑sensitivity. GenAC replaces the scalar output with a chain‑of‑thought that first thinks and then emits an integer 0‑10, which is normalized to a value. The prompt also supplies the actor’s size and current win‑rate, forcing the critic to condition on the actor’s state.

Two‑step training:

SFT on GPT‑5‑generated data.

Freeze the actor, train the critic with REINFORCE (GAE reduced to Monte‑Carlo return).

Experiments on six math benchmarks (DeepScaleR) show GenAC’s error decreasing with model size, stable across random seeds, and achieving the highest sample efficiency. When combined with PPO, GenAC outperforms GRPO, RLOO, VC‑PPO, and traditional critics, especially on out‑of‑distribution sets (AIME24, GPQA) where error reduction exceeds 50 %.

OPID: On‑Policy Skill Distillation for Agentic RL

OPID extracts global trajectory skills (overall workflow) and step‑level skills (critical decisions) from the model’s own on‑policy rollouts. A LLM‑based parser converts ordered execution logs into natural‑language skill descriptions. During training, the policy is routed:

If the current step matches a highlighted critical point, the step‑level skill is injected.

Otherwise, the global trajectory skill provides context.

The injected skill modifies token‑level probabilities; the difference between skill‑augmented and original logits forms a “skill‑distillation advantage”. This advantage is added to the usual group‑relative reward before PPO update.

Findings:

OPID consistently beats GRPO across model sizes, with larger gains on smaller bases.

Even without providing skills at inference time, OPID retains its advantage.

Using both skill tiers with proper routing outperforms naïve concatenation.

With only 60 % of data, OPID matches GRPO’s full‑data performance; at 80 % it surpasses it.

Overall Summary

GLM‑5.2 solves long‑trajectory compaction by moving from GRPO to token‑level critic‑based PPO and adding a two‑stage anti‑cheat monitor that intercepts only offending actions. Qwen’s verification framework combines unit tests, LLM judges, and human review, augmented by an Agentic Quality Judge, static rubric, and interactive judge to eliminate reward‑hacking across SWE, front‑end, and real‑world tasks. GenAC demonstrates that generative critics can scale and remain robust, delivering superior sample efficiency in RL. OPID shows that on‑policy skill extraction and routing provide consistent gains, especially for smaller models and low‑data regimes. Together, these advances illustrate a clear trend: verification systems must co‑evolve with increasingly capable agents, and sophisticated, multi‑level evaluation pipelines are essential for reliable Agentic RL.